Filtering and route lookup in a switching device

ABSTRACT

Methods and devices for processing packets are provided. The processing device may Include an input interface for receiving data units containing header information of respective packets; a first module configurable to perform packet filtering based on the received data units; a second module configurable to perform traffic analysis based on the received data units; a third module configurable to perform load balancing based on the received data units; and a fourth module configurable to perform route lookups based on the received data units.

RELATED APPLICATIONS

This is a continuation of U.S. patent application Ser. No. 10/017,719entitled “Filtering and Route Lookup in a Switching Device” for Fergusonet al. filed on Dec. 14, 2001 which is a continuation-in-part ofco-pending U.S. patent application Ser. No. 09/550,413 entitled“Filtering and Route Lookup in a Switching Device” for Ferguson et al.filed on Apr. 17, 2000, the content of which are hereby incorporated byreference.

Moreover, this application claims the benefit of the filing dates of thefollowing provisional patent applications for any inventions disclosedtherein in the manner provided by 35 U.S.C. §112, ¶1: U.S. ProvisionalApplication No. 60/258,882 entitled “Internet Processor” for Ferguson etal. filed on Jan. 2, 2001; U.S. Provisional Application No. 60/258,886entitled “Internet Processor” for Ferguson et al. filed on Jan. 2, 2001;U.S. Provisional Application No. 60/258,887 entitled “InternetProcessor” for Ferguson et al. filed on Jan. 2, 2001; and U.S.Provisional Application No. 60/258,888 entitled “Internet Processor” forFerguson et al. filed on Jan. 2, 2001. The contents of these provisionalpatent applications are hereby incorporated by reference.

BACKGROUND

The present invention relates generally to data routing systems, andmore particularly to methods and apparatus for efficiently routingpackets through a network.

In packet switched communication systems, a router is a switching devicewhich receives packets containing data or control information on oneport, and based on destination information contained within the packet,routes the packet out another port to the destination (or anintermediary destination).

Conventional routers perform this switching function by evaluatingheader information contained within a first data block in the packet inorder to determine the proper output port for a particular packet.

Efficient switching of packets through the router is of paramountconcern. Referring now to FIG. 1A, a conventional router includes aplurality of input ports 2 each including an input buffer (memory) 4, aswitching device 6 and a plurality of output ports S.

Data packets received at an input port 2 are stored, at leasttemporarily, in input buffer 4 while destination information associatedwith each packet is decoded to determine the appropriate switchingthrough the switching device 6. The size of input buffer 4 is based inpart on the speed with which the destination information may be decoded.If the decoding process takes too long as compared to the rate at whichpackets are received, large sized memory elements may be required orpackets may be dropped.

In addition, the size of input buffer may be influenced by a conditionreferred to as “blocking”. Packets may be forced to remain in the inputbuffer after the destination information is decoded if the switchingdevice cannot make the connection. Blocking refers to a condition inwhich a connection cannot be made in the switch due to theunavailability of the desired output port (the port is busy, e.g.,routing another packet from a different input port). In summary, thesize of input buffer 4 is dependent on a number of factors including theline input rate, the speed of the lookup process, and the blockingcharacteristics for the switching device. Unfortunately, conventionalrouters are inefficient in a number of respects. Each input portincludes a dedicated input buffer and memory sharing between input portsis not provided for in the design. Each input buffer must be sized tomeet the maximum throughput requirements for a given port. However,design trade-offs (cost) often necessitate smaller buffers for eachport. With the smaller buffers, the possibility arises for packets to bedropped due to blocking conditions. While excess memory capacitytypically exists in the router (due to the varied usage of the inputports), no means for taking advantage of the excess is afforded.

To minimize the occurrence of dropping packets, designers developed nonhead-of-line blocking routers. Referring now to FIG. 1B, a conventionalnon head-of-line blocking router includes a plurality of input ports 2each including an input buffer (memory) 4, a switching device 6 and aplurality of output ports 8 each having an output buffer (memory) 9. Inorder to provide non head-of-line blocking, each output port 8 isconfigured to include an output buffer so that each output port cansimultaneously be outputting packets as well as receiving new packetsfor output at a later time. As the size of the output buffer isincreased, fewer packets are dropped due to head-of line blocking atinput ports.

However, these designs are even more inefficient in terms of memorycapacity and cost. Again, each output port includes a dedicated outputbuffer and memory sharing between output ports is not provided for inthe design. Each output buffer must be sized to meet the maximumthroughput requirements for a given port (in order to maintain its nonhead-of-line blocking characteristics). Even more excess memory capacitytypically exists in the router (due to the varied usage of the inputports and output ports), yet no means for taking advantage of the excessis afforded. Twice the amount and bandwidth of memory has to be usedthan required to support the amount of data being moved through thesetypes of devices.

What is desirable is to produce a router where the data packets can flowto a common memory, while routing decisions are made off-line. Byseparating the data path, the path along which the packet data traversesthrough the router, and the control path, a path used in evaluating thepacket headers, memory can be conserved. In addition, by separating thedata and control path, advanced filtering, policing and other operationscan be performed without incurring expensive increases in the memoryrequirements for the router due to the additional time required toperform the extra operations.

SUMMARY OF THE INVENTION

In one aspect the invention provides a method for performing a lookup ina switching device of a packet switched network where the lookupincludes a plurality of distinct operations each of which returns aresult that includes a pointer to a next operation in a sequence ofoperations for the lookup. The method includes determining a firstlookup operation to be executed, executing the first lookup operationincluding returning a result and determining if the result includes apointer to another lookup operation in the sequence of operations. Ifthe result includes a pointer to another lookup operation, the lookupoperation indicated by the result is executed. Else, the lookup isterminated.

Aspects of the invention can include one or more of the followingfeatures. A lookup operation can be selected from the group of a treesearch, an index search and a filter. A lookup operation can include afunction list that specifies one or more functions to execute during theexecution of the lookup operation. The function can be selected from thegroup of a management function, accounting function and policingfunction. The method can include identifying when a lookup operationspecifies a function and executing the function including returning aresult that indicates a next lookup operation in the sequence ofoperations to be executed. The execution of the function includesdetermining when a packet should be sampled for further processing andincluding in the result a designation that indicates the packet is to besampled.

In another aspect the invention provides a method for performing alookup to determine routing for a packet through a switching device in apacket switched network. The method includes chaining a plurality oflookup operations in a sequence including linking each operation to asuccessive operation in the sequence such that an arbitrary sequence ofoperations can be specified to determine the routing of a packet throughthe switching device and executing the chain of lookup operations.

In another aspect the invention provides a method for performing alookup in a switching device. The method includes identifying a firstlookup operation in a sequence of lookup operations to be performed on apacket, executing the first lookup operation including returning aresult that is a pointer to a subsequent lookup operation in thesequence, executing the subsequent lookup including returning a resultthat is a pointer to a next lookup operation in the sequence, continuingto execute lookup operations in the sequence until a lookup operation inthe sequence returns a result that indicates that no more operations areto be processed and when a result indicates that no more operations areto be processed, returning a notification to the switching device thatincludes routing information for the routing of the packet through theswitching device.

In another aspect the invention provides a method for policing a streamin a switching device in a packet switched network. The method includes,in a single read operation, determining a data rate for the stream in atime interval and a policing decision for a current packet in the streamand, in a single write operation, writing the policy decision and countinformation for the stream without requiring global overhead to clearthe count at each time interval.

In another aspect the invention provides a method for updating a lookupdata structure in a lookup process. The lookup data structure includesan arbitrary sequence of lookup operations for determining the routingof a packet through a switching device in a packet switched network.Each lookup operation invokes a distinct lookup algorithm that calls adata structure that when executed returns a result that links to a nextlookup operation in the arbitrary sequence. The method includesdetermining a location in the sequence of lookup operations where anupdate is desired. If the update adds a lookup operation to the sequenceat the location, the added lookup operation is written to memory andlinked to a next lookup operation after the location. Thereafter, apointer in a lookup operation preceding the location is updated to pointto the added lookup operation. If the update deletes a lookup operationfrom the sequence at the location, a pointer in a lookup operationpreceding the location is updated to point to a next operation after thelocation and thereafter the lookup operation can be deleted from thememory.

In another aspect the invention provides a data structure for a lookupoperation. The lookup operation is in a sequence of lookup operationsthat, when executed by a switching device in a packet switched network,determines routing for a packet through the switching device. The packetincludes a key to be used in a lookup operation. The data structureincludes a next hop identifier for linking operations in an arbitrarysequence to determine the routing of the packet through the switchingdevice. The next hop identifier includes a pointer, an update and anoffset. The pointer points to a particular lookup operation selectedfrom a group of lookup operations. The update includes data for updatinga pointer that points to a starting byte in the key to be used in thelookup operation. The offset indicates an offset bit down from thestarting byte bit location to use for the lookup operation.

In another aspect the invention provides a method for performing alookup to determine routing for a packet through a switching device in apacket switched network. The method includes providing plural algorithmsin a lookup engine for performing distinct lookup operations, specifyingan arbitrary sequence of lookup operations to be performed when thepacket is received and executing lookup operations defined in thesequence in the order specified.

In another aspect the invention provides a route lookup engine forperforming a lookup in a packet switched network where the lookupincludes a plurality of distinct operations each of which returns aresult that includes a pointer to a next operation in a sequence ofoperations for the lookup. The apparatus includes one or more lookupoperation engines for executing lookup operations including returning aresult and a lookup engine The lookup engine is operable to determine afirst lookup operation in the sequence to be executed, evaluate theresult returned from the execution of the first lookup operation todetermine if the result includes a pointer to another lookup operationin the sequence of operations, invoke a particular lookup operationengine from the group of lookup operation engines based on the pointerto execute a next lookup operation in the sequence of operations andterminate the lookup and return a result to be used in routing thepacket through the packet switched network.

Aspects of the invention can include one or more of the followingfeatures. The lookup operation engines can be selected from the group ofa tree search look up engine, a index search index engine and a filterengine. The route lookup engine can include a memory configurable tostore one or more tree data structures and where the pointer returnedfor invoking the tree search engine includes an indicator pointing to aparticular tree data structure stored in the memory to be searched inthe lookup operation. The memory can include one or more index datastructures and where the pointer returned for invoking the index searchengine includes an indicator pointing to a particular index datastructure stored in the memory to be searched in the lookup operation.The memory can store one or more filter data structures and where thepointer returned for invoking the filter engine includes an indicatorpointing to a particular filter data structure stored in the memory tobe searched in the lookup operation.

A lookup operation can include a function list that specifies one ormore functions to execute during the execution of the lookup operationand where the lookup engine can be operable to read the function listand execute the one or more functions in the lookup. The function can beselected from the group of a management function, accounting functionand policing function. The lookup engine can be operable to identifywhen a lookup operation specifies a function and execute the functionincluding returning a result that indicates a next lookup operation inthe sequence of operations to be executed. The execution of the functioncan include determining when a packet should be sampled for furtherprocessing and including in the result a designation that indicates thepacket is to be sampled.

In another aspect the invention provides an apparatus for policing astream in a switching device in a packet switched network and includes abuffer for storing a count and a threshold for the stream and a policingengine. The policing engine is operable to, in a single read operation,determine a data rate for the stream in a time interval and a makepolicing decision for a current packet in the stream and, in a singlewrite operation, write count information for the stream after eachpacket in a stream is processed without requiring global overhead toclear the count at each time interval.

Aspects of the invention can include one or more of the followingfeatures. The buffer can include four values including a last timeadjustment value that is written in the single write operation toindicate a last time that the data rate was calculated, a current countvalue that indicates an amount of data that had been written as of thelast adjustment time, a threshold value that indicates the thresholdamount of data that can be passed in the stream before policing isrequired, and a credit value indicating the amount of counts to beapplied to the current count per unit time. The policing engine can beoperable to read, in the read operation, the four values and make thepolicing decision, and operable to write, in the write operation, a newvalue for the last time adjustment value and the current count valuethat reflects the processing of a current packet.

Aspects of the invention can include one or more of the followingadvantages. A technique is provided to implement traffic policing basedon a fixed window monitoring mechanism with a minimal use of memorybandwidth. A method and apparatus are provided for implementing ageneral purpose packet filter within a lookup engine for longest matchlookups. An apparatus is provided that supports chained lookupoperations. The apparatus includes a route lookup engine that includesplural engines each for performing a different type of lookup operation.An apparatus is provided to allow for the chaining of plural lookuptechniques in a switching device.

Other advantages and features will be apparent from the followingdescription and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are block diagrams of conventional router devices.

FIG. 2A is a schematic block diagram of a data routing system.

FIG. 2B is a schematic block diagram of a router.

FIG. 3 is a schematic block diagram of a multi-function port.

FIG. 4 is a schematic diagram showing the data transfers betweencomponents of the router of FIG. 2B.

FIG. 5 is a schematic block diagram of an input switch.

FIG. 6 is a schematic diagram of memory structure for the router.

FIG. 7 is a schematic diagram of the global memory for the router.

FIG. 8 is a schematic block diagram of a controller.

FIG. 9 shows a schematic block diagram for a key lookup engine.

FIG. 10 shows a packet processing operation.

FIG. 11 is a schematic block diagram of an output switch.

FIG. 12 is a schematic block diagram for an output section of amulti-function port.

FIG. 13 is a schematic block diagram for a queue system for storingnotifications.

FIG. 14 is a flow diagram for a process of routing a packet through arouter.

FIG. 15 is a graph of performance testing results.

FIG. 16 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 17 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 18 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 19 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 20 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 21 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 22 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 23 is a schematic block diagram of an exemplary loopback interface.

FIG. 24 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 25 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 26 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 27 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 28 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 29 illustrates an exemplary operation of a traffic shapingalgorithm.

FIG. 30 illustrates an exemplary operation of a traffic shapingalgorithm.

FIG. 31 illustrates exemplary soft traffic-policing function.

FIG. 32 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 33 illustrates an exemplary traffic policing operational model.

FIG. 34 shows an exemplary packet classification table.

FIG. 35 shows an exemplary packet notification record.

FIG. 36 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 37 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 38 shows an exemplary packet notification record.

FIG. 39 is a schematic block diagram of an exemplary networkconfiguration.

FIG. 40 is a schematic block diagram of an exemplary networkconfiguration.

DETAILED DESCRIPTION

Referring to FIG. 2A, in a packet switching system, a source 10 isconnected to one or more routers 20 for transmitting packets to one ormore destinations 30. Each router includes a plurality of multi-functionmultiports that are connected to various sources and destinations. Apacket from source 10 may pass through more than one router 20 prior toarriving at its destination.

Referring to FIG. 2B, each router 20 includes an input switch 100, anoutput switch 102, a global data buffer 104 including one or more memorybanks 105, a controller 106 and a plurality of multi-function multiports150 (150-0 through 150-3), respectively. Associated with the controller106 is a controller memory 109 for storing routing information. Inputswitch 100 and output switch 102 are connected to each multi-functionmultiport 150 in router 20. In one implementation, router 20 includesplug-and-play multi-function multiports which allow for easy expansioncapability. The present invention will be described with reference to asystem including eight multi-function multiports 150 (even though FIG.2B only shows four), with each multi-function multiport including up tosixteen input ports and sixteen output ports. Other configurations maybe used depending on user load conditions. Each multifunction multiportincludes one or more input ports, one or more output ports and a memory.The configuration and operation of the multi-function multiports will bedescribed in greater detail below.

In operation, packets are received at a multi-function multiport 150,transferred to input switch 100 and stored temporarily in global databuffer 104. When the packet is received by input switch 100, a key andother information is read from the packet and transferred (in the formof a notification) to controller 106. The key contains destinationinformation which is derived from the header field associated with thefirst block of data in a packet and other information (such as sourceID, priority data and flow ID).

A route lookup engine 110 in controller 106 performs a lookup based onthe notification information and returns a result which includes theoutput miultiport associated with the destination. The result is coupledwith other information (such as source ID, flow ID and packet length)for routing the packet through router 20 and provided as a notificationfrom controller 106 to output switch 102. Output switch 102 transfersthe notification to the identified multi-function multiport 150. Uponreceiving the notification information, the multi-function multiport 150initiates the transfer of the packet from global data buffer 104 throughoutput switch 102 to the appropriate multi-function multiport 150.

Multi-Function Multiports

Referring to FIG. 3, each multi-function multiport 150 includes an inputsection 270, an output section 280 and a memory section 290.

Input section 270 includes a line input interface 300, a data handler302 and an input switch interface 304. Output section 280 includes anoutput request processor 306, a line output interface 308, a storagedevice (memory) 310, stream output buffers 312 (one for each outputstream), output formatter 314, an output switch interface 316 and headand tail queue buffer 318. In addition, the output section includes aportion of input switch interface 304. Specifically, input switchinterface 304 includes read request queues 305, one for each memorybank. The use and operation of the read request queues, stream outputbuffers, and head and tail queue will be discussed in greater detailbelow.

Memory section 290 includes a memory bank 105 (which represents aportion of the global data buffer 104) and a notification area 319. Theuse an operation of the memory section will be discussed in greaterdetail below.

The multi-function multiport is used in conjunction with the inputswitch, output switch and controller as is shown in FIG. 4. The variouspiece components of the input section, output section and memory sectionare described in greater detail below. The combination of the devicesinto a single unit simplifies the interfaces between the components.

Referring again to FIG. 3, packets are received at line input interface300. As the packets are received, data handler 302 divides the packetsreceived into fixed lengths cells. In one implementation, the length ofeach cell is 80 bytes, with 16 bytes of internal header (controlinformation) and 64 bytes of cell data. As the data handler divides theincoming packets into fixed length cells, it synchronously outputs thecells to input switch 100 through input switch interface 304.

Each cell transferred from a multi-function multiport 150 to the inputswitch contains a cell header and cell data. The cell header can includea type field, stream field, and packet header fields. In addition, thecell header can include an independent read request in the form of amulti-function multiport identifier and address.

The type field indicates the type of cell to be transferred from themulti-function multiport. At each cell slot (20 clock cycles in oneimplementation), a multi-function multiport may transfer either a datacell, an indirect cell placeholder, or a delayed indirect cellplaceholder. Data cells contain data associated with an incoming packet.An indirect cell placeholder is an empty cell, and is used inconjunction with indirect addressing for the storage of the cells in theglobal data buffer 104. Delayed indirect cell placeholders arise when adata stream that requires indirect addressing terminates at a time priorto the designated time for writing the last indirect addressing cellassociated with the data stream to global data buffer 104. Thegeneration and operation of indirect placeholders and delayed indirectplaceholders will be discussed in greater detail below.

The stream field indicates the stream to which the cell data belongs. Inone implementation, each multi-function multiport is capable of handlingup to sixteen separate streams of data at a time, one on each of itsrespective 16 input ports.

The packet header field contains header information associated with agiven packet and includes start offset information, packet length andinterface index information.

The multi-function multiport identifier identifies the multi-functionmultiport which is sourcing the read request. The address indicates theaddress in global data buffer 104 to be read.

A single cell can be transferred from a multi-function multiport 150 toinput switch 100 at each cell (time) slot “T”. For a given cell slot“T”, input switch 100 receives a total of “N” cells, where “N” is equalto the number of multi-function multiports. Similarly, a single cell canbe transferred from the input switch 100 to memory 104, from the memory104 to the output switch 102, and finally from the output switch 102 toa multi-function multiport 150 at each cell (time) slot “T” as is shownin FIG. 4.

In one implementation, cells from a given stream may be written tomemory in an order that is different from the arrival order. These outof order writes are performed to make efficient use of scarce bandwidthbetween the multi-function multiports and the input switch. When apacket comes in to the multi-function multiport, it is broken up intocells as the bytes arrive and the cells are placed in per-bank outputqueues on the way to the input switch. These queues are designed toshare scarce interconnect bandwidth between the streams of amulti-functional multiport in the most efficient way possible, but theyhave the detrimental effect of reordering cells at the interface betweenthe multi-function inultiport and the input switch. Thus the cells froma given stream may arrive at the input switch out of order. Themulti-function multiport marks the data cells of a stream with one offour codes: first cell (FC); intermediate data cell (DC); last cell(LC); or first cell which happens to be also a last cell (FLC).

Input Switch

Referring to FIGS. 2B and 5, input switch 100 includes a round robindata handler 500, one or more input interfaces (501-0 through 501-7, onefor each multi-function multiport 150), one or more memory interfaces502 (502-0 through 502-7, one associated with each memory bank), a likeplurality of pointers 504 (504-0 through 504-7), an output processor505, one or more output interfaces 506 (506-0 through 506-7, one foreach multi-function multiport 150), a reservation table 508, an indirectcell processor 510, controller interface 512 and read controller 517.

a) Transfers Through the Input Switch

Round robin data handler 500 receives cells from each multi-functionmultiport and transfers them to output processor 505 for output to anappropriate memory bank 105 in global data buffer 104. Round robin datahandler 500 services the inputs (cells) received on input interfaces 501in a round robin, time division multiplexed manner. That is, for a givencell slot, one cell from each multi-function multiport is received atthe round robin data handler 500 and subsequently transferred to outputprocessor 505 for transfer at the next cell slot to a memory bank 105 inglobal data buffer 104. At the next time cell slot, data handler 500transfers the next cell received from the same multi-function multiportto output processor 505 for transfer to a different memory bank. In oneimplementation, the next cell received is transferred to the next memorybank (next in numerical order, modulo N) in the memory array.Alternatively, another time dependent permutation may be used to controlthe transfer of successive cells from the same multi-function multiport.

Round robin data handler 500 and output processor 505 within the inputswitch 100 transfer cells out to global data buffer 104 on transmissionlines. Output processor 505 outputs one cell to each memory bank in asingle cell slot. One cell from each multifunction multiport is writtento global data buffer 104 per cell slot. Round robin data handler 500time division multiplexes the transfers to output processor 505 suchthat consecutive cells from the same multi-function multiport arewritten to consecutive memory banks 105 (modulo N) in global data buffer104.

Pointer 504 indicates the location in an associated memory bank to whichthe next cell will be written. Output processor 505 writes a cell to amemory location in a particular memory bank based on the next availableaddress in the bank as is indicated by the associated pointer 504.

b) Key Reading and the Linking Process

Round robin data handler 500 includes a key reading engine 514 fordetermining the key information associated with a first cell in a packetand a linking engine 515 for linking cells in the same packet.

The process of reading key information is known in the art. After thekey is determined for a given packet, it is stored temporarily in keybuffer 516 in input switch 100 until the entire packet has been storedin global data buffer 104. Each entry in the key buffer is referred toas a notification or “info cell” and includes a key, full address,offsets and an indirect cell indicator and can include otherinformation.

Linking engine 515 determines the starting address (full address) inmemory for where the first cell in a given packet is to be stored inmemory. The starting address includes the bank number in global databuffer 104 (the bank number which is assigned to store the cell by roundrobin data handler 500) and the first available address location in thedesignated bank (as is indicated by the associated pointer 504). Thestarting address is stored in key buffer 516 along with the associatedkey for the packet. When the next cell associated with the same packetarrives at switch 100, an offset associated with the offset at which thecell is to be written (relative to the full address) is computed andstored in key buffer 516. In one implementation, up to four offsets arestored. Each offset address is computed based on the relative offset inmemory between the location of the last cell in memory and the value ofthe pointer 504 associated with the current memory bank which is to bewritten.

If more than five data cells are included in a packet, then the indirectcell indicator for that packet is set, and the last offset indicates theaddress in memory where the first indirect cell associated with thepacket is stored. Indirect cells are described in greater detail belowand in copending application entitled “Separation of Data and Control ina Switching Device” filed Dec. 17, 1999 and assigned U.S. patentapplication Ser. No. 09/466,864, the contents of which are expresslyincorporated herein by reference.

After the packet has been stored in memory, the associated notificationin key buffer 516 (a route lookup request) is forwarded through thecontroller interface 512 to the controller 106 for processing.Alternatively, the notification may be transferred after the first fivecells have been stored in memory.

As described above, the data cells are stored in the global buffer uponreceipt. The data path for the data packets flows directly from theinput port on which a packet is received (the multi-function multiport150) to the global data buffer 104. The data packets remain in theglobal data buffer 104 while a routing decision is made in a separatecontrol path using controller 106. The separation of the data path andcontrol path allows for the sharing of the memory resources among all ofthe input ports.

The linking or threading of cells for a packet is performed by using theoffsets described above and indirect cells. Offsets are used to linkcells in a packet. Offsets may be stored along with key information androuted through controller 106 (FIG. 2B) or may be stored in indirectcells. In one implementation, if a cell contains 5 cells or less, noindirect cells are required to be used. Indirect cell processor 510performs the linking of cells in memory for a given packet. Indirectcell processor 510 generates indirect cells for storage in global databuffer 104. Indirect cells contain offset information associated withthe relative offset in memory space between contiguous cells in thepacket. Indirect cell processor includes indirect cell memory 520 forstoring indirect cell data during the formation of indirect cells.

As was described above, when a packet is received, the linking engineprocesses the first five cells and stores linking information in theform of a start address and four offsets in key buffer 516. In the eventmore than five cells are contained within a packet, the indirect cellprocessor takes over for the linking engine and computes the offsetsassociated with the locations in memory where the remaining cells in thepacket are stored. Round robin processor 500 passes cells to the outputprocessor 505 for transfer to an associated memory bank in global databuffer 104. Round robin processor 500 enables the indirect cellprocessor when the packet being processed contains more than 5 cells(based on header information included within the first cell). At thetime for writing the fifth cell to memory, indirect cell processor 510stores in indirect cell memory 520 the address (the “indirect celladdress”) associated with the location in memory at which the fifth cellwould have been written if it had been the last cell in the packet. Theindirect cell address indicates the location in memory where theindirect cell is to be written when full (or when the last cell of thepacket is processed).

When an indirect cell is full (having stored offsets in all availablelocations except the last field), then the indirect cell processorstores the offset associated with the location in memory where the nextindirect cell is located. Thereafter, the full indirect cell is writtento its appropriate place in memory. The writing of the indirect cell tomemory coincides with the receipt of an indirect cell placeholder by theinput switch 100 from the associated multi-function multiport 150. Thisprocess continues until the last cell in a packet is stored in memory.At that time, the last indirect cell is written to memory, and theassociated entry from the key buffer 516 is transferred to thecontroller 106 for processing. For a given packet, all indirect cellsare written to the same memory bank in the global memory buffer.

As often will be the case, the last cell of a packet will not coincidewith the timing required to write the completed indirect cellimmediately into memory. This is because packet length is completelyarbitrary. The end of a packet will likely not coincide with the lastavailable entry of an indirect cell. When a packet has completed (allcells have been received by the input switch) and a last entry in theindirect cell is written, the indirect cell is free to be written tomemory. However, the writing will be delayed until the proper time,hence the term delayed indirect cell. A delayed indirect cell is aindirect cell that is the last indirect cell associated with a packet.It is delayed, because it is written to memory after the rest of thepacket has been written to memory. The timing of the write to memory isdictated by the address which is reserved for the indirect cell. As wasdescribed above, at the time for the creation of an indirect cell, itsposition in memory is reserved. The delayed indirect cell will bewritten to memory at the next time slot available for the particularmulti-function multiport to write to the particular memory bank afterthe packet has been completed. The timing of the write to memory ofdelayed indirect cells coincides with the receipt of a delayed indirectplaceholder from the appropriate multi-function multiport 150.

c) Transfers to Memory

At each cell slot, output processor 505 generates a cell that includes aread request source field, read address, write address and data field(cell data received from multiport 150). The read request source fieldindicates the output port (in the particular multi-function multiport150) requesting the read (destination output port). Output processor 505receives read requests from read controller 517 and bundles the readrequest with any write request received from round robin data handler500 destined for the same memory bank. At each cell slot, outputprocessor 505 provides a cell which may include a write and read requestto each memory bank 105 in global data buffer 104.

Read controller 517 controls the transfer of read request signalsflowing from input switch 100 out memory interface 502 to the individualmemory banks in global data buffer 104. Read controller 517 receivesread requests from each multi-function multiport through outputinterfaces 506. The format of each request includes sourceidentification (output port) and a full address in memory which is to beread. At each cell slot, each multifunction multiport port may generatea read request for processing by switch 100 to read a memory location inglobal data buffer 104, resulting in the reading of a cell (a readreply) from a memory bank 15 (on a subsequent cell slot) to outputswitch 102.

Read controller 517 loads a reservation table 508 as requests totransfer packets are received from the various multi-function multiports150. The reservation table is loaded such that at every cell slot asingle read request is generated for each bank of memory 105. Thestructure of the reservation table is described in greater detail in“Separation of Data and Control in a Switching Device”. At each cellslot, each multi-function multiport is capable of requesting a read froma single memory bank 105 in global data buffer 104. Associated withreservation table 508 is a read pointer. The pointer points to a nextrow in the reservation table to be read. Rows ahead of the read pointercorrespond to requests that will be queued at a later cell slot time. Inone implementation, the pointer moves at least one row in each cell slottime.

Memory Architecture

Referring now to FIG. 6, main memory 104 is used as temporary bufferstorage for packets flowing into the system on input streams 1052 andout of the system on output streams 1054. Main memory is divided intotwo distinct parts: a global data buffer 104 that is used to storeincoming packets while one or more lookup engines in the controller 106determine the outgoing stream for each packet; and packet notificationqueues 319 that are used to store packet pointers (notifications) afterthe outgoing stream has been determined. Notification queues 319 areassociated with outgoing streams, whereas the global data buffer 104forms a common pool shared amongst all the streams.

Referring now to FIG. 7, main memory includes a plurality of memorybanks. Associated with each memory bank is an input switch interface (aninput port) 304 and output switch interface (an output port) 316. Ateach cell slot, each memory bank receives at most one write and one readrequest via input switch interface 304. The write requests areassociated with cells received from a multi-function multiport 150. Readrequests reflect a request for cell data to be transferred from a memorybank to output switch 102 for ultimate transfer to a requestingmulti-function multiport 150.

The memory in the multi-function multiport configuration is physicallydistributed across a number of banks b, one bank for each activemulti-function multiport in the system. Each bank is divided into twocontiguous, non-overlapping regions referred to as global data area 105and the notification area (notification queues 319). The global dataarea for a bank constitutes 1/b of the memory of the global data buffer104. The notification area provides space for queuing notifications thatwill be sent out on the line output interface 308 for a givenmulti-function multiport. Typically, the global data area is four timeslarger than the notification area; this factor derives from the ratiobetween data size and notification size for the shortest packet.

In one implementation, each bank's memory bandwidth is sufficient forreading and writing packets from a full-duplex OC-48 interface as wellas for queuing and dequeuing notifications for the worst-case example ofsingle-cell packets. Thus, both the aggregate memory size and theaggregate memory bandwidth scale linearly with the number of activemulti-function multiports b in the system.

In one implementation, each memory bank is implemented as two sub-banksusing two 72-bit wide SDRAM (static dynamic random access memory) DIMM's(dynamic in-line memory modules) cycling at 125 MHZ. The sub-banks aretransparent to the input and output switch resulting in what appears tobe one continuous bank from the perspective of the switches. However,the sub-bank architecture allows for better throughput. Each DIMM has a72-bit wide ECC (error correction code) protected data path going to 9SDRAM chips each of which is 8 bits wide. The two DIMM's have separateaddress busses and are addressed independently of one another. TheDIMM's are interleaved on bit 0 of the 23-bit address. In oneimplementation, the smallest memory bank configuration is 32 MBytes,using 16 Mbit chips and the largest is 512 MBytes, using 256 Mbit chips.

As was described above, a bank can receive at most one read request andone write request every cell slot. Since a cell slot is 20 clock cyclesat 125 MHZ, this works out to a peak bandwidth demand of 400 MBytes/secfor reads and 400 MBytes/sec for writes. The worst case notificationload occurs for single cell packets. For unicast traffic, this load isexactly ¼ the data bandwidth which works out to 100 MBytes/sec for readsand 100 MBytes/sec for writes. In this implementation, the total peakmemory bandwidth needed is therefore 1 GByte/sec.

In this implementation, the peak transfer rate of each DIMM is 1GByte/sec, but the sustained rate depends on the actual mix of reads andwrites and how the addresses are distributed over the internal DIMMbanks. In practice, each DIMM is expected to deliver a sustained datarate of around 650 MBytes/sec. The total of 1.3 GBytes/sec supplied bythe two groups is 30% larger than the maximum sustained requirement of 1GByte/sec. The 30% headroom provides a way to sustain instantaneousloads where one DIMM has more references directed to it than the other.The memory controller for the two DIMM's resides in the multi-functionmultiport.

In one implementation, all banks are made the same size andapproximately ⅕th of the memory in each bank is allocated to thenotification area and ⅘th to the global data area. The purpose of thisallocation is to make it exceedingly unlikely for a stream to run out ofmemory because of space in its notification queue. With a worst casepacket size of 64 bytes, notifications (sized at 16 bytes) need ¼th theamount of storage that packet data needs, which is exactly theproportion allocated. Any cell in the global data buffer may be accessedvia its physical cell pointer, which identifies the physical bank numberand the address of the cell within the bank. The physical cell pointerdefines a system-wide physical address space. To simplify addresscomputations, as well as to provide a mechanism to detect old packets,accesses to the global packet buffer are performed through a system-widevirtual address space that maps to the physical address space.

Incoming packets are broken up into as many cells as needed and thecells are written to the global packet data buffer as they arrive asdescribed above. The global data buffer is treated as a single largecircular buffer. The input switch maintains an array of write pointers,one per active bank, to keep track of where to write the next cell. Thepointers start out at one end of the buffer and advance until theyeventually wrap around and overwrite packets that were written a longtime ago. An ageing mechanism is used to guard against reading cellsthat may have been overwritten by subsequent packets. The cells ofpackets arriving on a given stream are interleaved strictly across theactive banks to spread the bandwidth load.

Controller

Referring now to FIG. 8, controller 106 includes controller memory 109,route lookup engine 110, input switch interface 800 and output switchinterface 802. Controller 106 receives a route lookup request from inputswitch 100 at the input switch interface 800. In one implementation, aplurality of route lookup engines 110 are included in controller 106,each receiving lookup requests in round-robin fashion so as to speed therouting process. In one implementation, controller memory 109 is afour-bank static random access memory (SRAM) that requires thirty sixroute lookup engines 110 to service at full bandwidth.

The present invention is scalable with respect to performance. That is,the number of route lookup engines 110 included within the controllermay be increased to provide higher performance without requiring anincrease in memory size. In one implementation, the number of routelookup engines is nines times as great as the number of memory banks incontroller memory 109. Alternatively, lesser cost and performance unitsmay use lesser numbers of route lookup engines 110 or more engines asrequired.

a) Controller Operation

Referring to FIGS. 2B, 3 and 8, in operation, packets are received at aninput port 150, transferred to input switch 100 and stored temporarilyin memory 104. When the packet is received by switch 100, a keyextraction engine reads the key from the packet and transfers the keyand other information (the notification) to controller 106. The inputswitch also includes a transfer engine for transferring packets receivedfrom an input port 150 to memory 104.

The key includes at least destination information and may also includesource information, a flow identifier and physical source information(input port ID). The key can be located in the header field associatedwith the first block of data in a packet. The header may contain otherinformation (ISO layer 2 and layer 3 headers), such information ispassed to memory for storage. The process of reading key informationfrom a packet is known in the art. The present invention accommodateskeys of various types. For example, keys for various protocols may bedesignated (IPV4, IPV6, etc.). The length of the key is user definable.In general, the key is derived from the header, but portions may also bederived from the payload (data field associated with the packet).

When the controller receives the notification information, it mustdetermine a key type. In one implementation, a plurality of key typesare defined. The user may define up to 4 types of keys, each havingvariable length. The key type can be defined by a two bit field in theheader. A lookup of the two bit field is used to determine anappropriate starting hop (as described below).

Thereafter, an assigned route lookup engine 110 performs a lookup forthe notification. The lookup can include a plurality of chained lookupoperations, one of which can be a jtree search. A jtree (jtrie) is adata structure that is used to locate the best (longest) matching routefor a given key. At the completion of the lookup, the route lookupengine returns a result which includes the output port associated withthe destination. The result and other information (source ID, flow ID,packet length, quality of service and statistical information) forrouting the packet through the router combine to form a resultnotification. The result notification is transferred from the controller106 to the output switch 102. Upon receiving the result notification,the output switch 102 initiates the transfer of the packet from memory104 to the respective output port 150 associated with the result.

In one implementation, the data structure for the result notificationincludes a destination mask, a next hop index pointer, full address,offsets and packet length. The destination mask is used to indicatewhich multi-function multiport connected to output switch 102 is totransfer the packet. In one implementation, the result notification maybe sent to more than one multi-function multiport resulting in thebroadcast of the associated packet. Associated with each multi-functionmultiport 150 is a storage 310. The next hop index pointer points to alocation in storage (memory) 310. Storage 310 is used to store mediaheader information associated with a particular type of packet transfer.Next hop addresses, media headers and storage 310 will be described ingreater detail below in association with the output section ofmulti-function multiport 150. The full address indicates the startingaddress in the global data buffer where the first cell in the packet isstored. As was described above, offsets provide linking information forretrieving cells or an indirect cell associated with the packet. Thepacket length indicates the length of the associated packet and may beused to determine if indirect cells will have to be retrieved.

b) Route Lookup Engine

Each route lookup engine performs packet (key) processing. Packetprocessing is the process of examining the contents of a packet headerand performing functions such as route lookup, filtering, or flowpolicing based on the values of fields in the header. The result ofpacket processing determines how a packet should be forwarded in therouter.

Referring now to FIG. 9, each route lookup engine 110 includes a keybuffer 902, a result buffer 904, a key engine 905, one or morespecialized engines for processing packets 906, a current key pointer908 and starting hop table 910. In one implementation, each route lookupengine 110 includes a general purpose key engine 905 and pluralspecialized engines 906. The general purpose key engine 905 receives thekey from the input switch, loads the key and result buffers, performsinitializations, unloads the buffers and performs other operations insupport of the lookup process. Specialized engines operate oninstructions or data structures stored in memory 920 to perform aparticular function. Functions can be selected from lookup operations,filtering, policing, management or other functions. In oneimplementation, the specialized engines can be selected from the groupof a firewall engine 906 a, a policing engine 906 b, index engine 906 cand trie search engine 906 d. Each of these engines can be invoked toperform an operation and assist in determining a forwarding decision fora packet. As will be described below, more than one engine can beinvoked to operate on each packet.

Key engine 905 stores the fields from a packet that have been selectedto be part of the key for packet processing in key buffer 902. Any partof a packet can be selected to be part of the key, depending on theapplication. The key extraction process is completed in the input switch102 as described above. The results of the key extraction process (theextracted key) and other information forms the notification that ispassed to the controller 106. Any part of the notification can beextracted by the key engine 905 and written to the key buffer 902. A‘key’ can consist of two parts. In one implementation, the first eightbytes of the key are constructed either from the contents of thenotification for the packet, or built from intermediate results of routelookups. The remaining bytes of the key, which are variable in length upto 41 bytes, are those extracted from the payload of the packet. In oneimplementation, key buffer 902 is a 64 byte buffer, physically locatedin the first 8 double words in the key engine's memory (not shown).Fixed data is stored in bytes 0-7 of the buffer while the variable keydata is stored in bytes 8 and beyond.

The first 2 bytes of the key buffer are used as an overwrite area;various intermediate next hops may write data in this area to beaccessed by subsequent lookup stages. These two bytes are initialized tozero. The first 4 bits of the second word are also used as an overwritearea for sampling indications. Sampling refers to a process offorwarding a copy of a packet to an external system for processing. Apacket that is designated to be sampled is switched by the router asdefined in the notification, but a copy of the packet (or portion of thepacket) is created and forwarded to a system for further processing. Thelookup process executed by the route lookup engine may include thedesignation of a packet for sampling. The further processing can includea management function that can be provided either on or, more typically,off the router. Packets can be designated to be forwarded to themanagement function for analysis. For example, a sampling of all of thepackets that are from a particular source can be sent to the managementfunction for further analysis. The sampling bits can be set to designatea packet as requiring further processing. In this way, when the (result)notification for the packet is processed, a copy of the packet (orportion of the packet) can be forwarded to the management function forfurther analysis. The sampling bits can be set and modified in thelookup process. As such, whether a particular packet is a candidate fora sampling operation can be decided based on a lookup result. In oneimplementation, the low order bit (sn[0]) is initialized to the value ofan incoming sampling (SN) bit in the notification from the input switch,and the other three bits are initialized to zero. In this way, a deviceupstream from the controller (e.g., the input switch or themultifunction multiport) can designate some of the packets for sampling.In one implementation, the sampling bits are a mask for a predefinednumber of sampling operations. That is, the setting of a bit in thesample bits indicates a sampling operation to be performed.Alternatively, the sampling bits can be a set to indicate that aparticular packet is a candidate for sampling. The decision as towhether or not the candidate is actually sampled can be made outside thelookup engine based on a programmable probability algorithm.

As various algorithms process the key, the key buffer can be used tocommunicate from one processing step to another. More specifically, thekey buffer can include one or more rewrite bytes. The rewrite bytes forma data area that can be used by one processing step in a lookup (onelookup operation) to directly pass data to another processing step. Inone implementation, key buffer 902 includes 2 rewrite bytes.

In one implementation, the format of the fixed and variable areas in thekey buffer 902 is as shown in Table 1-1.

TABLE 1-1 Key Buffer Format

Associated with key buffer 902 is the current key pointer 908 thatpoints to the location in the key buffer 902 that should be treated asthe beginning of the key in the current processing step. The current keypointer 908 can be moved from field to field in the key buffer 902 fordifferent lookups that are chained together. By default, at the start ofa lookup, the current key pointer points to the start of the variablelength key. The current key pointer 908 is maintained by key engine 905.

The key engine 905 stores the result of packet processing which isinformation on how the packet should be forwarded in the result buffer904. When multiple lookups are performed for the same packet, the resultmay be modified at the end of each lookup. The contents of the resultbuffer 904 at the end of the last lookup for a packet is the finalresult. In one implementation, the contents of the result buffer are asshown in Table 1-2.

TABLE 1-2 Result Buffer Format

where:

-   -   n=next hop set bit. If n is 1, some next hop has caused the next        hop index and destination mask (dest_mask) fields to be set. If        n is zero the next hop has not been set and the dest_mask is        effectively zero.    -   s_PR=priority set bit. Defaults to zero. If s_PR is 1, some next        hop has caused the PR bits to be overwritten.    -   PR=the priority bits. Set by default to the priority bits in the        incoming notification. May be modified by next hops with the        s_PR bit set.    -   s_(x)=‘x’ set bit. Defaults to zero. If s_x is 1, some next hop        has caused the ‘x’ bit to be overwritten.    -   x=‘extra’ bit. Set by default to the P[2] bit in the incoming        notification. May be modified by next hops with the s_x bit set.

Starting hop table 910 includes starting hops for each packet type. Thestarting hop table 920 can be shared by all of the route lookup engines110 in controller 106. Entries in the starting hop table 920 can be ofthe form of a final next hop or an intermediate next hop. Next hop datastructures and the processing of next hops are described in greaterdetail below.

c) Packet Processing

Packets are processed in accordance with a next hop instruction. A nexthop is a data structure stored in memory that either contains the finalresult of packet processing for a packet or acts as a link to anotherlookup for the same packet. The key engine 905 receives a next hopinstruction, and either processes the instruction directly, or invokes aspecialized engine within the route lookup engine to process the nexthop. A “final next hop” contains information on the final destination ofthe packet. An “intermediate next hop” is a link to the next lookup stepand contains a command specifying the lookup algorithm type, a memoryaddress pointer to the beginning of the lookup data structure, and anoptional pointer offset for moving the current key pointer to adifferent key buffer field. An “extended next hop” is a special type ofintermediate next hop that contains a memory address pointer to a listof instructions (more than one next hop can be pointing to the samelist). These instructions specify whether to modify certain fields inthe key buffer and may place lookup results in the result buffer. Thelist can end with an intermediate next hop, a final next hop or withouta next hop (where no next hop is specified, the lookup process ends andthe current contents of the result buffer are used as the result). A“starting next hop” is the next hop specifying the first lookup step fora packet and is either of the form of a final next hop or anintermediate next hop. Starting next hops are stored in the starting hoptable 910. The data structure for the next hops is described in greaterdetail below.

Referring now to FIGS. 9 and 10, a method 1000 for packet processing isshown. Packets are processed by the key engine 905 in four steps:initiate, lookup, link, and terminate.

In the initiate step 1002, the key buffer 902 is loaded withpre-selected fields from the notification 1004. The result buffer 904and current key pointer 908 are initialized to a default values 1006. Apre-selected field from the packet is then used to index the startinghop table 910 1008. Different starting next hops correspond toindependent packet processing paths. The protocol type of a packet canbe used as the index for selecting the “starting next hop” so that eachprotocol can be processed in different ways.

Once the starting next hop is obtained, the link portion 1024 of theprocess begins. The link portion 1024 of the process includes threechecks. Each of the checks evaluates a current next hop. By current nexthop we refer to either the starting next hop, intermediate next hopreturned from a lookup operation or a result returned after processingan extended next hop. First, a check is made to determine if the currentnext hop (the starting next hop, intermediate next hop or result) is anextended next hop 1020. If the next hop is an extended next hop, thenthe function specified by the extended next hop (e.g., policing,sampling, counting or other function) is executed 1022. The contents ofthe key buffer and the result buffer may be modified before the nextlookup step is performed. By allowing the modification of the contentsof the key buffer 902, a subsequent lookup operation can use the resultsfrom an earlier lookup step as part of its key. By allowing themodification of the contents of the result buffer 904, intermediateresults can be stored. If not modified by a subsequent lookup step, theintermediate result, or some fields from it, may eventually form thefinal result. At the completion of the execution of the associatedfunction and the modification of the buffers, a result is returned(1023). The result can be in the form of a next hop. Thereafter, theprocess continues back at step 1020.

In the second check of the link portion 1024, the current next hop isevaluated to determine if it is of the form of an intermediate next hop1016. The check can be performed after the first check performed in step1020 fails (i.e., the current next hop is not an extended next hop). Ifthe current next hop is an intermediate next hop, then the processcontinues at step 1018 where the current key pointer is set to thelocation specified by the intermediate next hop and a lookup isperformed on the packet. The intermediate next hop acts as a linkbetween two lookups. The intermediate next hop specifies the type of thenext lookup (e.g., lookup engine 906 to invoke), the memory location ofthe lookup data structure (e.g. index table, jtree, firewall filterprogram), and the new location of the current key pointer 908.

After the current key pointer is set to the new location in step 1018(as required), the lookup portion 1014 of the process is invoked. In oneimplementation, the lookup portion can include the invocation of one ormore specialized engines in the route lookup engine 110. In the lookupportion 1014, the key (or portion of the key or other notification data)to be operated on is extracted from the key buffer (1010) and aspecified lookup operation is executed (1012). The lookup operation maygo on for any number of clock cycles and any number of memory referencesuntil a result, in the form of a next hop, is obtained 1012. Thereafter,the process continues at step 1020.

If the current next hop returned is not an intermediate next hop, thethird check of the link process 1024 is invoked. More specifically, ifthe check in step 1016 determines that the current next hop is not anintermediate next hop, then a check is made to determine if the currentnext hop is a final next hop (1017). This completes the linking portion1024.

After the check in step 1017 is performed the terminate portion 1030 ofthe process is invoked. In the terminate portion, packet processing forthe current packet is terminated and a result is returned. Morespecifically, if the current next hop is not a final next hop theprocess continues at step 1028. If the current next hop is a final nexthop, then the process continues at step 1026 where any information inthe current next hop relating to the final destination of the packet canbe copied into the appropriate fields in the result buffer (1026), asrequired. The contents of the result buffer is then unloaded (1028) andused to forward the current packet through the router. Thereafter theprocess ends.

d) Processing Algorithms

The basic routing in a IP (internet protocol) network is done based on alongest match lookup on a field of bits in the packet header. To enhancethis routing and make it more intelligent, the system adds processingfeatures to be able to filter the packets based on some fields in thepacket header. Enhancements have been added to allow some accountingability and by also providing flow control based on a policing engine.

Packet processing includes processing the fields of a packet header,sometimes also known as a key, to perform the required functions likeroute lookup, filtering or flow policing. Key information for a packetcan be processed using several different algorithms to generate aresultant notification (result) which is then used to forward the datapacket appropriately.

In one implementation, three base algorithms can be selected from forpacket processing and include an index table lookup, variable lengthbest match lookup (i.e., a jtree lookup) and a firewall lookup. Each ofthe algorithms uses a next hop data structure to initiate processing. Atthe end of each processing step (after invoking an algorithm to operateon a designated portion of the key or executing a specified function)the result is also a data structure in the form of a next hop. Based onthe variety of nexthops one can initiate new processing steps or end theoverall packet processing function. The next hops thus form the primarydata structure that can be used to initiate a lookup, chain differentlookups and terminate a lookup. In addition, the next hop data structurealso include provisions for supporting a variety of added features likepacket counting, packet sampling and flow based policing of packets.

1) Index Engine

An index table lookup is performed by invoking the index engine 906 c toextract the specified bits of index from the key and add the specifiedbits to a base pointer to compute the address of a next hop to read (seebelow for next hop data structure). Associated with the index engine 906c are one or more index tables 916 stored in a memory 920. Memory 920includes the various data structures operated on by the various enginesin the route lookup engine 110. In one implementation, memory 920 isseparate from the route lookup engine 110 and can be accessed through amemory interface 911. In the implementation shown, memory 920 includes asingle index table 916 but plural index tables can be included in memory920. The key engine 905 reads a current next hop, and if the lookupcalls for an index search, invokes the index engine 906 c to perform anindex lookup on a specified index table 916. An index search next hopincludes a designator for the position in the key to use in the indexsearch and the index table 916 to use.

In one implementation, an index table 916 includes a variable number ofdoublewords (at least one) and has a starting address located at adoubleword boundary. The first word of the index table 916 contains thetable size and description of the location (relative to the current keypointer) and number of the key bits to be extracted as an index. Thesecond word of the index table 916 holds the default next hop, which isread if the index exceeds the size of the index table 916. The baseaddress of the index table 916, to which the index is added to computean address to read if no overflow occurs, is implicitly the address ofthe third word (i.e. second doubleword) of the table. Each entry in anindex table 916 includes a one word ‘next hop’. The memory allocationfor routes is rounded up to a doubleword boundary, but otherwise is theminimum necessary to hold the route information. An index table 916 canbe defined to be a multiple of two words long in contiguous memory. Inone implementation, the format of the start of an index table is shownin Table 1-3.

TABLE 1-3 Index Table Format

where:

-   -   the i_off field is a bit offset from the location of the current        key pointer to the first bit of the index. The value can be set        to 0-7 inclusive, which allows the start of the index to be        anywhere in the byte the current key pointer points to.    -   the idx_nbits field indicates the number of bits that should be        extracted to form the index. Valid values are 0-22 inclusive. A        value of 0 in this field causes the default next hop to be        unconditionally read. A value larger than 22 is invalid and        causes the notification to be discarded.    -   The number of pairs of next hops in the table (exclusive of the        default next hop) is indicated by the table size field. If the        index extracted is index[21:0], the address of the (descriptor        word of the) index table is table_address, and the table size in        doublewords is table_descriptor[31:11], the address from which        the next hop result is extracted is computed as:        -   if (index[21:1]>=table_description[31:11]) then            -   next_hop_address=table_address+1;        -   else            -   next_hop_address=table_address+2+index.

2) Trie Search Engine (i.e., Variable Length Best Match Lookup)

If the current next hop indicates a tree based search is to beperformed, the key engine 905 invokes the trie search engine 906 d toperform a lookup operation that includes a longest match lookuptraversing a radix trie data structure (referred to herein as a“jtree”). The search is based on the specified number of bits at aparticular starting point in the key buffer. The process for performingthe longest best match lookup is described in greater detail incopending application “Separation of Data and Control in a SwitchingDevice”. The result of the longest match lookup of the key bits is anext hop. More specifically, a route in a jtree consists of a one word‘next hop’, at a double-word aligned memory location, followed by zeroor more words of prefix information. One or more jtrees 914 are storedin memory 920. A next hop specifying a jtree search includes identifyinginformation for the particular jtree to be searched. The storage of ajtree in memory 920 is described in greater detail in “Separation ofData and Control in a Switching Device”.

3) Firewall Engine

The firewall engine 906 a is an instruction-based, protocol-independentmatch engine which operates on data in the key buffer. When a next hopspecifies a firewall lookup, the key engine 905 invokes the firewallengine 906 a which in turn retrieves a firewall filter program frommemory 920. The program includes a stream of one or more instructionsthat can be executed to perform the requested firewall services. Mostinstructions are ‘match’ instructions, that is, the instruction takessome data from the key buffer and matches it to the instruction'soperands. A ‘true’ match causes the next instruction in the stream to beexecuted, a ‘false’ match causes a forward branch to be taken. A byteindex to the data item in the key buffer to be matched, and an offsetfor the branch forward, is included in the instruction word. The matchdata is included in the operands.

The other instruction type is an ‘action’ instruction. Actioninstructions may carry some fields unique to the instruction in theremainder of the 32-bit word, but typically don't have operands. Someaction instructions terminate the search and indicate the result toreturn in the form of a next hop structure. The firewall engine 906 aand firewall filter programs are described in greater detail below.

e) Next Hop

There are several ‘next hop’ formats. One of these is a “final” nexthop, which terminates the search and contains a final result for thelookup. The others are “intermediate” next hops, which indicate how afurther search should be performed.

1) Final Next Hop

In one implementation, bit 10 of the next hop is the ‘final’ bit. If thefinal bit is set, the next hop is in ‘final’ format, otherwise it is oneof the ‘intermediate’ formats. In one implementation, the format for afinal hop is as shown in Table 1-4.

TABLE 1-4 Final Next Hop Format

where:

-   -   f=final bit. set to 1 for final next hops, set to 0 otherwise    -   m=multicast bit. If set, and if this next hop is attached to a        route in a jtree, a 14-bit ‘interface index’ is appended to the        route prefix data which must be matched against the incoming        interface index in the key buffer. If there is a mismatch the        packet is discarded. The m-bit is ignored in index table        lookups.    -   a=accounting bit. If set, and if this next hop is attached to a        route in a jtree, a three word packet+byte counter pair is        appended to the prefix (and multicast interface index, if        present) data. These counters must be incremented to account for        packets and bytes matching this route. This bit is ignored in        index table lookups.    -   s=set bit for the x and PR bits. If ‘s’ is 1, x and PR are        copied into the corresponding fields in the result buffer. If        ‘s’ is 0, x and PR in the result buffer remain unchanged. The        outgoing notification will have the final values of x and PR in        the result buffer. Note that {x, PR} in the result buffer        default to the incoming priority bits {P[2], P[1:0]}, so the        outgoing notification will have the incoming priority bits if no        next hop modifies them.    -   PR=new priority bits. If ‘s’ is 1, the corresponding field in        the result buffer will be overwritten by PR. As a result, the        outgoing notification will have these new priority bits. The        priority bits are not changed if ‘s’ is 0.    -   x=new ‘extra’ bit. If ‘s’ is 1, the corresponding field in the        result buffer will be overwritten by ‘x’. As a result, the        outgoing notification will have this new ‘x’ bit. ‘X’ bit is not        changed if ‘s’ is 0. The hardware picks up this ‘x’ bit and        forwards it to the output port.    -   the destination mask (dest_mask) field is a 9-bit mask        indicating where the packet should be forwarded to. Bits 0        through 7, when set, indicate that the packet should be sent to        physical banks 0 through 7, inclusive. When bit 8 is set the        packet is locally destined.    -   the ‘next hop index’ is a 16-bit value used by other parts of        the forwarding system to determine outgoing processing of the        packet.

The execution of the final next hop causes the destination mask and nexthop index fields to be copied into the result buffer. The lookup processthen ends and key engine 905 uploads the latest results in the resultbuffer including forming an outgoing notification that includes theresults. In one implementation, packet filtering (filtering or policing)can be performed based on the data included in the final next hop. Forexample, in one implementation, if the destination mask in the resultbuffer is set to a value of 9'h0 at the end of a lookup, the packet isdiscarded and a discard counter (CF_DBR_CNTR) is incremented. Final nexthops cause the lookup to end, so final next hops with dest_mask=9'h0will always cause the packet to be dropped.

2) Intermediate Next Hops

All other next hops are ‘intermediate format’ next hops (hereinafter,intermediate next hop). Intermediate next hops can be chained to allowfor plural lookup operations to be performed on a single packet. Forexample, a packet can be filtered, subject to a jtree lookup todetermine a route, then subsequently filtered again prior to determininga final routing decision. The three lookup operations (filter, jtreesearch, filter) can be specified by chaining next hops. The process forchaining next hops is described in greater detail below. Oneimplementation for the format of an intermediate next hop is as shown inTable 1-5.

TABLE 1-5 Intermediate Next Hop Format

where

fid is the format identifier (ID) for the intermediate next hop. In oneimplementation, there are 5 lid's including:

-   -   0—jtree lookup    -   1—firewall/index table lookup    -   2—jtree/index table ID lookup    -   3—multiple next hops    -   4—extended next hop

In one implementation, if an undefined intermediate next hop (e.g.,fid=3′h5, 3′h6, 3′h7) is encountered, the lookup will end immediately,the packet is dropped, and a discard counter (CF_DBSFT_CNTR counter) isincremented.

a) Jtree Lookup Intermediate Next Hop (fid=0)

In one implementation, a jtree lookup has an intermediate next hopformat as is shown in Table 1-6.

TABLE 1-6 Jtree Lookup Intermediate next hop format

where:

-   -   bits [31:6] of the next hop, with bits [5:0] hardwired to zero,        is a tree pointer to the root of a jtree to be searched,    -   the byte_offset is the offset to change the current key pointer        by. The value of the byte_offset field is added to the current        key pointer modulus 64. That is, a byte_offset of 0 will keep        the current key pointer unchanged, a byte_offset of 1 would move        the current key pointer to the right (i.e. forward) one byte in        the key, and a byte_offset of 63 would move the current key        pointer one byte left (i.e. backward) in the key. If the current        key pointer is moved to a location beyond the end of the key        delivered from the input switch, an error occurs (equivalent to        a truncated key error when doing tree lookups). The packet is        dropped, and a discard counter (CF_DTK_CNTR discard counter) is        incremented.

b) Firewall/Index Table Lookup Intermediate Next Hop (fid=1)

In one implementation, a firewall/index table lookup has a next hopformat as is shown in Table 1-7.

TABLE 1-7 Firewall/Index Table Intermediate Next Hop Format

where:

-   -   i=index table bit. If this bit is 1, the next lookup to be done        is an index table lookup. If this bit is 0 the next lookup to be        done is a firewall filter program.    -   bits [31:10] of the next hop is the word address of the first        word of the firewall filter program or index table, depending on        the setting of the ‘i’ bit.    -   the byte_offset is the offset to change the current key pointer        by.

c) Jtree/Index Table ID Lookup (fid=2)

In one implementation, a jtree/index table ID lookup has a next hopformat as is shown in Table 1-8.

TABLE 1-8 Jtree/Index Table ID Intermediate Next Hop Format

where:

-   -   i=index table lookup. If the ‘i’ bit is set to 1, the lookup        that is done next is an index table lookup. The table_id refers        to the entry in the 64-entry on-chip index table directory from        which to get the index table descriptor and start the lookup. If        the ‘i’ bit is set to zero, the lookup that is done next is a        jtree lookup. The table_id refers to the on-chip jump (for        table_id 0 or 1 and jump table enabled) or tid table (table_id 2        to 63 or jump table not enabled) entry from which to start the        lookup.    -   table_id is a 6 bit index. If the ‘i’ bit is set to 1, table_id        is the index table descriptor pointing to the index table into        which the search will be continued. If the ‘i’ bit is set to        zero, table_id is the jtree root pointer pointing to the tree        into which the search will be continued.    -   byte_offset is the offset to the current key pointer (i.e. as in        the fid=0 intermediate next hop above).    -   s_x=set bit for the ‘x’ bit. If s_x is 1, ‘x’ is copied into the        corresponding field in the result buffer. If s_x is 0, ‘x’ in        the result buffer remains unchanged.    -   x=extra bit. Copied into the corresponding field in the result        buffer if s_x is set.    -   s_PR=set bit for the PR bits. If s_PR is 1, PR is copied into        the corresponding field in the result buffer. If s_PR is 0, PR        in the result buffer remains unchanged.    -   PR=the priority bits. Copied into the corresponding field in the        result buffer if s_PR is set.    -   o=set bit for the rewrite field. If ‘o’ is 1, the rewrite byte        in location [31:24] is copied into one of the first 2 bytes of        the key, with the rb (i.e. rewrite byte) field specifying which        byte is rewritten.    -   rb=specifies whether rewrite byte 0 (rb=0) or rewrite byte 1        (rb=1) should be overwritten if the ‘o’ bit is set.

The rewrite, x, and PR updates take effect before the jtree or indextable ID lookup begin. For example, if {rb, o}={1, 1}, and byte_offsetmoves the current key pointer to point to rewrite1, the key buffer isupdated with the new rewrite byte in the next hop, and then the fid=2lookup is performed using the new rewrite byte value as the key.

d) Multiple Next Hop Intermediate Next Hop (fid=3)

A multiple next hop can be used to perform load balancing operations.Multiple next hops can be designated to alleviate loading conditions forany particular hop. In one implementation, the multiple next hops arelisted, and a selection can be made based on loading or otherconsiderations. In one implementation, the router can be configured fora balanced, unbalanced or incoming interface mode of operation.

In balanced mode, the particular next hop is randomly selected from thedesignated group of next hop candidates. In one implementation, theselection is made by taking a hash of the key and based on the hashedvalue, assigning the packet to a particular one of the candidate nexthops. Alternatively, the selection can be made by selecting a randomnumber or by pseudo random means that take into account history or otherconsiderations in making the determination.

In unbalanced mode, weights can be assigned to individual candidate nexthops, and a weight affected decision can be made to select a next hopfrom the candidate next hops. The weights can be assigned to supportcertain quality of service or priority service considerations.

The incoming interface mode can be used to screen packets depending onthe type of next hop being processed. A multiple next hop includes alist that specifies candidate hops. The incoming interface mode can beused to select among the candidates. More specifically, if the incominginterface designator for a packet matches the incoming interfacedesignator associated with a determined route, a first next hop in themultiple next hop list is selected as the next hop to be used inforwarding the packet. If the designators do not match, the second entryin the list can be selected as the next hop to be used in forwarding thepacket. For all other next hop types, if the match bit (m-bit describedbelow) is set and no match is detected, then the packet can be droppedand an appropriate discard counter can be set.

One implementation of a multiple next hop format is as shown in Table1-9.

TABLE 1-9 Multiple Next Hop Intermediate Next Hop Format

where:

-   -   m=multicast bit. If set, and if this next hop is attached to a        route in a jtree, a 14-bit ‘interface index’ is appended to the        route prefix data which must be matched against the incoming        interface index in the key buffer. If there is a mismatch the        packet is discarded unless the ‘h’ bit is set (see below). The        m-bit is ignored in index table lookups.    -   a=accounting bit. If set, and if this next hop is attached to a        route in a jtree, a three word packet+byte counter pair is        appended to the prefix (and multicast interface index, if        present) data. These counters must be incremented to account for        packets and bytes matching this route. The a-bit is ignored in        index table lookups.    -   bits [31:10] of the next hop form the memory address of the        first entry in a list of next hops for this route, any of which        may be used to forward the packet.    -   #hops has a value between 1 and 15 inclusive, or 0. If the value        is 0, the number of next hops is 16, otherwise the number of        next hops is #hops. If #hops is 1, there is only 1 next hop, so        the hardware will always choose the same next hop, regardless of        the mode or the hash value.    -   h=hash bit. Along with the ‘m’ bit, indicates how the hash value        is reduced to the index of a next hop in the list. The modes can        be selected from a balanced mode (m=X and h=0), an unbalanced        mode (m=0 and h=1) and an incoming interface mode (iif mode)        (m=1 and h=1). If the multiple next hop is read from a data        structure that does not support multicast, the m-bit is        interpreted as 0 even if it is set. For example, if a multiple        next hop with {m, h}={1, 1} is read from an index table,        unbalanced mode processing is performed, not iif mode. In the        iif mode, there must be at least 2 next hops in the multiple        next hops list.

e) Extended Next Hop Intermediate Next Hop (fid=4)

The extended next hop format allows the lookup process to implement oneor more functions. Associated with an extended next hop is a extendednext hop list that includes one or more functions to be executed. Thefunctions can include policing operations. One implementation for anextended next hop format is as shown in Table 1-10.

TABLE 1-10 Extended Next Hop Intermediate Next Hop Format

where:

-   -   m=multicast bit. If set, and if this next hop is attached to a        route in a jtree, a 14-bit ‘interface index’ is appended to the        route prefix data which must be matched against the incoming        interface index in the key buffer. If there is a mismatch the        packet is discarded. The m-bit is ignored in index table        lookups.    -   a=accounting bit. If set, and if this next hop is attached to a        route in a jtree, a three word packet+byte counter pair is        appended to the prefix (and multicast interface index, if        present) data. These counters must be incremented to account for        packets and bytes matching this route. The a-bit is ignored in        index table lookups.    -   bits [31:10] of the next hop form the memory address of the        first entry in the extended next hop list. The F, O, CI, I, and        CA bits determine what is located in the extended next hop list.    -   F=modified final bit. If this bit is set, the word pointed to by        the address contains a modified final next hop, which causes        results to be set into the result register.    -   O=overwrite bit. If this bit is set, the extended next hop list        includes an overwrite word, which includes data to be inserted        in the rewrite bytes in the key, as well as settings for the        precedence bits. The overwrite next hop is included after the        modified final next hop, or at the start of the list if the F        bit is clear.    -   CI=counter indirect bit. If set, the next hop list includes a        32-bit counter pointer. This pointer contains information about        the location of a counter and its type. If present this pointer        is located after the modified final and overwrite words.    -   I=intermediate bit. If set, the last word entry in the list is        an intermediate next hop of one of the formats described        previously. If no intermediate next hop is included in the list        the lookup terminates. If the I-bit is not set, the lookup        terminates immediately and the outgoing notification is built        based on the current results in the result buffer. Software can        terminate a lookup using an extended next hop with the I-bit not        set or even none of the F, O, CI, I, CA bits set. In one        implementation, the I-bit can be set and a final next hop can be        put in the extended next hop list. Note that the modified final        next hop is for placing a lookup result in result buffer to be        used when the lookup ends. For example, a result can be put in        result buffer before an output filter program is run. If the        packet is not dropped by firewall filter, the lookup result        stored in the result buffer will be used.    -   CA=counter attached bit. If set, the next hop list includes an        attached, standard, byte-and-packet counter. The counter is        attached to the end of the list, possibly with a single pad word        to align the counter to an odd-word boundary.

f) Policing

In networking systems, there is a need to be able to count the data bitsin a flow (flow is a logical unit of data transmitted for one place toanother). Additionally it is also useful to be able to measure the rateof the data bits (in say bits per second) and further be able to shapethe data flow as per requirements. The data flows can be shaped to becapped at a maximum bandwidth usage or otherwise restricted (e.g., aftera bandwidth usage has been achieved, reduce the priority of service forthe flow).

A number of terms are used in this section and are defined as follows.“Data Rate” as used herein refers to the measure of data transferred ina specified duration of time and is typically measured in bits persecond (bps, or Kbps, Mbps, Gbps). “Accounting” as used herein refers tothe ability to count the number of bits per flow. “Rate Measure” as usedherein refers to the ability to measure the data rate of a flow.“Policing” as used herein refers to the ability to measure the rate andthen shape it to a specified threshold.

In conventional systems, policing operations include two steps:measurement and flow policing. For a stream of data, a conventionalsystem needed to have a counter which keeps a count of the number ofbits of data in that stream that have been transferred. Additionally, atimer was needed which was used to set a data sampling interval. Giventhe number of data bits and a time reference, say 100 microseconds, aconventional system could count the number of data bits in that time andmultiply by, say 10000, to get a bits per second data rate measure forthe stream. If there are thousands of such streams monitored by thesystem, a large memory may be required to store the count informationfor the data bits.

In the monitoring step, for each chunk of data per stream, the systemwould read the counter and increment it with the number of bits in apacket. So for each packet transferred for a stream, the system wouldneed to read the memory once and write to it once. Additionally ifduring this continuous process of counting the bits

per stream, the system needed to measure the data rate for individuallypolicing each flow, then an additional read of a threshold-count valueis required every time the data counter is updated. Furthermore, thiscounter ideally is cleared every time interval so that the rate can bemeasured over a next time interval. Thus for rate measurements to makepolicing decisions for a flow, an additional read (every counter update)and a write (every time interval) to memory may be required. Assuming atime interval of 100 microseconds and have 10000 streams, then there isa fixed overhead of 100 million writes to memory per second required insuch a system. The actual number of reads and writes to memory forcounting the data bits per stream and the read for the threshold-countvalue varies with the number of packets/cells transferred in a timeinterval for each particular stream.

In addition, the time interval and the total number of streams determinethe fixed overhead required for the computations. The smaller (finer)the time interval the more accurate the rate measurement would behowever the proportion of fixed overhead bandwidth also increases. Theoverall memory bandwidth requirement is proportional to the fixedoverhead and the rate of packet/cell transfers in the system. Thedrawbacks of this type of measurement technique is that one needs asignificant memory bandwidth overhead. This requirement grows as thetime interval reduces and the number of streams increases. The number ofaccesses required to count the data bits per packet/cell for each streamdepends on the number of packets/cells transferred. This is verydifficult to predict and thus any technique used for data ratecomputations will need to adapt to this need.

In the present system, a new approach is proposed. The approach is basedon the observation that for policing a flow a system needs to make amemory reference to update the counts every time a packet/cell flowsthru the device (i.e. router, switch). In a relatively busy system,where memory accesses are precious, the assumption is that in a set timeinterval there is a very high probability that each stream would have atleast one packet/cell transfer per unit time interval. In such anenvironment the goal is to be able to compute the data bits, data rateand make a decision on policing each flow in a burst read and writeoperation to memory without the need for any global overhead to clearall the counters each time interval. This can be accomplished by saving(in memory) the last access time (referred to herein as last adjustmenttime) together with the data bit count for every flow. The policingdecision can be made by computing how many bits of data are allowed inthe time interval between the arrival of two packets belonging to thesame flow, triggering access to the same counter.

More specifically, a policing function can be implemented as part of alookup chain by designating an intermediate next hop that includes anextended next hop format having a list that includes a counter function.The counter can be updated by the policing engine as described below.Depending on the updates, policing decisions can be made on a packet bypacket basis. A single burst read operation can be performed to read allof the data required to make the policing decision [the threshold countvalue (credit_limit), the data counter value (current_credit), the lasttime updated (last_adjustment_time), the amount to be updated per unittime (time_credit)].

The last time updated reflects the time of arrival of the last packetreceived for a given flow. A single burst write operation is performedto update the counter with a new count value and the time of arrival forthe last packet processed. In the present system, the counters are notupdated at a fixed time interval, and instead are updated only whentraffic is received in the given flow. The burst operations result in alarge savings on memory bandwidth in the system.

In one implementation, the format for a policing counter for use in anextended next hop is shown in Table 1-11.

TABLE 1-11 Policing Counter Format

where:

-   -   last_adjustment_time indicates the last time this policing        counter was incremented. The timestamp is referenced to one of        two on-chip global time counters, with the particular counter        selected by the setting of the ‘r’ bit in the counter pointer        data structure pointed to by an extended next hop. In one        implementation, the high rate policing counter increments once        every 8.192 us, the low rate every 262.144 us.    -   time_credit indicates the amount of credit this counter receives        per time increment. The units are as specified in the        counter-pointer units field.    -   credit_limit indicates the limit to which the current_credit        field is allowed to increase. Units are as specified in the        counter-pointer units field, times 16.    -   current_credit indicates the amount of credit currently        accumulated.

An algorithm for updating the counter is as follows. The algorithm canbe executed by the policing engine 906 b in the route lookup engine 110.The policing engine 906 b receives packet length from the notification,and rate and units information from the key engine 905. The packetlength is first adjusted by rounding to units specified in the counterpointer data structure pointed to by an extended next hop, i.e.

-   -   if (units==2′b00) {        -   adj_packet_length=packet_length;    -   } else if (units=2′b01) {        -   adj_packet_length=(packet_length+2)>>2;    -   } else if (units 2′b10) {        -   adj_packet_length=(packet_length+8)>>4;    -   } else {        -   adj_packet_length=(packet_length+32)>>6;    -   }

The adjustment can be performed before the first word of the counter isread. When the first word of the counter is read, then the new credit iscomputed based on the last adjustment time and the current time, wherethe latter is selected by the rate bit. The new credit computation isdefined by: new_credit=time_credit*(current_time−last_adjustment_time).In this implementation, the new credit value gets the lower 18 bits ofthe result of the multiplication. If the upper 14 bits of the result arenon-zero then the notification is considered to be within specificationand the current credit value is updated with the credit limit ratherthan the equation below. If the upper 14 bits are zero then thecomputations below should be performed to check if the notification willbe out of specification and how the data structure needs to be updated.

When the second word arrives the boolean policing result can be computedas: out_of_spec=(adj_packet_length>(current_credit+new_credit)); wherethe add must maintain 19 bit precision. The values written back to thelocations in the counter are:

-   -   time_credit=time_credit;    -   last_adjustment_time=current_time;    -   credit_limit=credit_limit;    -   if (out_of_spec) {        -   temp=current_credit+new_credit;    -   } else {        -   temp=current_credit+new_credit−adj_packet_length;    -   }    -   current_credit=min({credit_limit, 4′h0}, temp); where any        out_of_spec result is returned to the key engine for packet        disposal.

g) Firewall Filter

The firewall engine provides a filter by running an instruction engineover a portion of data extracted from the key buffer. For IP packetsthis data can include interface class, IP option flags, incominginterface, fragment offset, destination address, source address,protocol, source port, destination port and tcp flags.

A filter program includes a stream of instructions. The filter programis produced by a user and subsequently compiled to form executableinstructions that are stored in a memory (i.e., memory 920 of FIG. 9).The filter is executed by a firewall engine (i.e., firewall engine 906a) in accordance with the processing of a next hop. In oneimplementation, the user can create a filter program using two basictypes of instructions: match instructions and action instructions. Eachbasic instruction has a value to compare to a data quantity, andoccasionally a mask (defaults to 0) to bit-fiddle the data beforecomparison. The match instructions are described in greater detailbelow. A branch can be taken when the comparison is true or false.

Each (logical) interface family (i.e., incoming interface in the routingdevice) may (or may not) have a filter program designated for incomingpackets, a separate filter program for outgoing packets, and aconfigurable ‘interface class’ byte for use in choosing output filtersbased on the incoming interface for a packet.

The structure of a filter program is as follows. Each filter programincludes a set of rules. Each rule has a set of zero or more matchconditions, and an action which is taken if all match conditions aretrue. Rules are logically executed in the order they appear in thefilter, with the action of the first matching rule being executed.

An example of an IP packet filter produced by a user is shown in Table1-12 below.

TABLE 1-12 IP Filter filter 3 ip {  rule 5 {   protocol udp,50-82 ;  action count accept ;  }  rule 10 {   protocol tcp,udp,50-82 ;  source-port 7-64,512-777 ;   destination-port 121-153 ;   optionslsrr|ssrr ;   from 192.168/17 ;   to 128.100/16 ;   tcp-flags (syn &!ack) #same as ‘setup’    | (rst | ack);# same as ‘established’  fragment-flags (df | mf); # also ‘dont-fragment’ # and‘more-fragments’   fragment-offset 1-8191; # same as ‘fragment’   actioncount log reject host;  }  rule 20 {   protocol icmp;   icmptype 1-52;  action discard;  }  rule 30 {   action accept count;  } }

Rule numbering is provided to allow the user to identify either a ruleitself (for modifications), or its relative ordering with respect toother rules (for additions).

In one implementation, matches can be of plural kinds. Matches onnumeric fields (e.g. protocol,

port numbers, fragment offset, icmp type) can specify a separate list ofnumeric ranges. Matches on bit fields (e.g. tcpflags, options,fragment-flags) can specify a set of bits combined with booleanoperators. Matches on addresses are used to match prefixes. Some matchconditions can have abbreviations (e.g. ‘established’ for ‘tcp-flagsrstlack’).

The result of a successful match is to take the ‘action’. The action maybe one of ‘accept’, ‘discard’ or ‘reject’, with modifiers ‘count’ and/or‘log’. ‘Count’ counts the number of matches for a given rule, while‘log’ can be used to sample a packet for further processing. The logaction can include the setting of sample bits in the result buffer. Thesetting of sample bits can result in the copying of a packet to anexternal processor for evaluation in addition to the routing of thepacket to its intended destination. Sampling is described in greaterdetail above.

Each filter program is provided to a compiler. The compiler can beseparate from the route lookup engine. The compiler generates the finalinstructions which can be executed by the firewall engine. The firewallengine can be implemented in a software or hardware engine.

The compiler compiles each rule in the order it appears in the filterprogram, but can be configured to change the ordering of match conditionevaluations to suit its own purposes. The compiler evaluates the matchconditions one by one in the order it has decided upon, branchingforward into another rule when a match evaluates false. When a match isfound false, however, the filter engine knows that (1) all prior matchconditions in the rule were true, while (2) the failing match conditionwas false. The compiler can reorganize the match order to make use ofthis type of information. For example, the compiler can skip over anyimmediately subsequent rules that cannot match (i.e. if ‘protocol tcp;’fails, all subsequent rules with ‘protocol tcp;’ will also fail and maybe skipped) and as far into the list of match conditions to a firstfeasible subsequent rule. The compiler can then remove dead matchconditions before compiling to instruction code. This skip over featureallows the compiler to generate better instruction variants. Forexample, a set of rules with identical match conditions except for thesource prefix, for example, can be compiled into a single evaluation ofthe common match conditions plus a tree lookup on the source address.

1. Instructions

As described above, the firewall engine is an instruction-based,protocol-independent match engine inside each route lookup engine thatoperates on the data in the 64-byte key buffer. In one implementation,each ‘instruction’ is a 4-byte quantity followed by zero or more 4-byteoperands. Most instructions are ‘match’ instructions, that is they takesome data from the key buffer and match it to their operands. A ‘true’match causes the next instruction in the stream to be executed, a‘false’ match causes a forward branch to be taken. A byte index to thedata item in the key buffer to be matched, and an offset for the branchforward, is included in the instruction word. The match data is includedin the operands. The other instruction type is an ‘action’ instruction.Action instructions may carry some fields unique to the instruction inthe remainder of the 32-bit word, but usually don't include operands.Some action instructions terminate the search and indicate the result toreturn.

The firewall engine includes a current data pointer (not shown) that isa byte pointer pointing to the data in the 64-byte key buffer to beoperated on by the firewall engine. The current data pointer isindependent of the current key pointer used outside of firewallprograms. In fact, the current key pointer remains at a fixed locationduring firewall program processing. When a firewall intermediate nexthop (fid=1, i=0) is encountered and control is passed to the firewallengine for executing the indicated filter program, the current datapointer is initialized to the value of the current key pointer (currentkey pointer value is the value after adding byte offset specified in theintermediate next hop). The current data pointer is updated every timean instruction containing the data offset field is executed. The dataoffset is an absolute byte address, not an offset from the currentvalue. This means that the current data pointer simply takes on thevalue of the data offset.

The data offset points to the data in the key buffer to be used for thecurrent instruction (in the filter program). It is the byte offset fromthe beginning of the key buffer, not an offset from the current datapointer value. For instructions that do not have the data offset field,the current data offset, i.e. the current data pointer value is used asthe data offset. In one implementation, the branch offsets in firewallinstructions are self-referential word offsets. The firewall engineincludes a program counter which keeps track of the 22-bit word addressof the current instruction word being executed. The next instruction tobe executed after a branch is set to be: the program counter (22-bitword addr)+1+branch offset (word offset, variable number of bits). Byexpressing all memory references in terms of an offset relative to theprevious location read, the firewall program can be independent of theposition into which it is placed in memory.

In one implementation, there are eight ‘match’ operations supported. Theoperation (op) field contains the unique operation code (opcode)indicating the type of match operation to be performed. In oneimplementations, each match operation is performed on 1-, 2- or4-byte-wide data. The operation expects to operate on ‘data’ (that isdata extracted at some offset into the key buffer), a ‘value’ (a valueextracted from the instruction) and a ‘mask’ (a second value sometimesincluded with the instruction where the default is all-zeroes). In oneimplementation, the eight operations include: an equivalence operation,a greater than operation, exclusive “or”, exclusive “and”,non-equivalence operation, less than, exclusive “nor” and exclusive“nand” operations.

a) One-Byte Match Instructions

One-Byte Match with Mask

Byte-wide instructions are identified by a ‘01’ in the low order bits.The first of these carries both data and a mask, but uses the currentdata offset.

where ‘value’ and ‘mask’ are included in the instruction and ‘data’ isthe byte of data at the current data offset.One-Byte Match without Mask

This second form of operation explicitly sets the data offset, but usesa default mask.

where ‘value’ is extracted from the instruction and ‘mask’ is all zeroes(by default). The current data offset is set from the instruction, and‘data’ is extracted from the byte at that offset.

b) Two-Byte Match Instructions

Two-Byte Match with Mask

In a two byte match operation, the first byte match sets the data offsetand both a ‘value’ and ‘mask’. Note that the branch offset is actuallyan offset from the word after the second instruction word.

c) Four-Byte Match Instructions

Finally, there are 4 byte instructions. For these the ‘value’ and ‘mask’words, if any, are always appended in subsequent instructions.

Four-Byte Match without Mask

In this format, the branch offset is actually an offset from the wordafter the second instruction word.

Four-Byte Match with Mask

Here the branch offset is actually an offset from the word after thethird instruction word.

h) Action Instructions

The remaining instructions do not include a numeric match (as requiredby each of the matching instructions set forth above). As such, they arespecial instructions in the firewall instruction set.

Long Branch Instruction Format

The canonical format for a long branch instruction is:

This instruction sets the data offset to that specified, and fetches thenext instruction from ‘branch offset’ words past the next word. Notethat setting the ‘branch offset’ to zero provides a method to reset the‘data offset’ without branching.

Termination Action Instructions

There are 4 different regular termination instructions. They aredistinguished by a field (the “res” field) in the instruction. Theformat for a regular termination instruction is:

-   -   res=2′b00—Discard notification. The lookup is terminated        immediately and the packet is discarded. The CF_DBR_CNTR (a        32-bit discard counter) is incremented. (If the corresponding        discard diagnostic bit is set, a copy of the notification is        sent to the host.) Note that the packet is discarded regardless        of whether the OP or TE (TE enabled) bit is set.    -   res=2′b01—Send to Host. The lookup is terminated immediately and        a notification is sent to the host by setting the destination        mask (dest_mask=9′h100). The next_hop_index in the outgoing        notification is set to the value of the next_hop_index field in        the termination instruction. PR[1:0] is set to 2′b00 (i.e. low        priority) regardless of its current value in the result buffer.        The x field is not changed, i.e. goes out with its current        result buffer value. The {rewrite, rb, o} fields in the        instruction are used to modify the corresponding rewrite1 field        in the key buffer which gets copied into the outgoing        notification. The res=2′b01 termination can be used as a        firewall “reject” instruction and the rewrite1 byte can be the        “reject code”.    -   res=2′b10—Next lookup is a jtree lookup (fid=2, i=0) in the        jtree specified by table_id. Byte_offset is used to set the        current key pointer for the next lookup. The {s_x, x} and {s_PR,        PR} fields are used to modify the corresponding priority bits in        the result register.    -   res=2′b11—Next lookup is an index table lookup (fid=2, i=1) in        the table specified by table_id. Byte_offset is used to set the        current key pointer for the next lookup. The {s_x, x} and {s_PR,        PR} fields are used to modify the corresponding priority bits in        the result register.

Regular Termination res2′b00 Instruction Format

Regular Termination res=2′b01 Instruction Format

where,

-   -   o=set bit for the rewrite field. If ‘o’ is 1, the rewrite byte        in location [15:8] is copied into one of the first 2 bytes in        the key buffer, with the rb (i.e. rewrite byte) field specifying        which byte is rewritten.    -   rb=specifies whether rewrite byte 0 (rb=0) or rewrite byte 1        (rb=1) should be overwritten if the ‘o’ bit is set.

Regular Termination res2′b10, 2′b11 Instruction Format

-   -   o1=set bit for the rewrite1 field. If ‘o1’ is 1, the rewriter        byte in location [31:24] is copied into the first byte in the        key buffer.        Extended Next Hop Termination Instruction Format

An extended next hop termination instruction can invoke an extended nexthop instruction set as described above. The branch offset to an extendednext hop is used to compute the address of an extended next hop list.The extended next hop list is located at: program counter (22-bit wordaddr)+1+branch offset to extended next hop list (22-bit word offset).Note that an extended next hop list linked to a firewall instruction maynot be double word aligned because the branch offset is a word offset.The CA, I, CI, O, and F bits are as in the jtree/index table extendednext hop format. The extended next hop termination with the I-bit notset can be used as a firewall “accept” instruction. The outgoingnotification will be built from the current contents of the resultbuffer. The result buffer should already have picked up the finalnext_hop_index and dest_mask from a route encountered before enteringthe output firewall filter. Note that an “accept” instruction may simplybe an extended next hop termination with none of the 5 control bits set.If the I-bit is set, and the corresponding next hop in the extended nexthop list is a final next hop, the result_buffer is updated and thelookup terminates normally.

An extended next hop termination instruction looks like:

Tree Search Action Instructions

A tree search action instruction invokes a longest match operation onthe field in the key buffer pointed to by the data offset using a jtreestored in memory 920. A tree search instruction can have the followingform:

Data offset sets the current data pointer prior to starting the treesearch. The instruction word immediately following the word indicated bythe current data pointer (can be any firewall instruction) is executedif no match is found during the tree lookup. No match here means thelookup falls off the top of the stack, i.e. stack underflow. Theremaining fields are used to form a jtree pointer which points at thenext tree node at: program counter (22-bit word addr)+1+branch offset tojtree (11-bit word offset). The jtree lookup performed is identical to a‘normal’ jtree lookup, with the following exceptions. The 21-bit ‘8-bytedouble-word memory address’ in any jtree pointer is a branch offsetrather than an absolute address. This keeps firewall filter programsindependent of memory location. The address of the next tree node iscalculated like this: program counter (22-bit word addr)+1+branch offset(21-bit word offset). Note that since the tree search instruction andjtree pointers in firewall programs have branch offsets that are wordoffsets, tree nodes in firewall jtrees may not be double word aligned.In one implementation, multicast iif comparisons and route accounting isnot done. The ‘next hop’ in a firewall jtree must be one of a longbranch instruction, a regular termination instruction (any of the 4 restypes) or an extended next hop termination instruction. The ‘next hop’is restricted to these instructions because the word in memoryimmediately following it is the prefix. If the ‘next hop’ is not one ofthe allowed instructions, the packet is discarded as a bad softwarediscard, and CF_DBSFT_CNTR (a 16-bit discard counter) is incremented. Ifa corresponding discard diagnostic bit is set, a copy of thenotification is sent to the host.

Tree Search Fail Instruction

The tree search fail instruction allows firewall jtrees to have prefixeswhose next hop means ‘this search failed, execute the instruction afterthe tree search instruction’. In one implementation, the failinstruction is actually a tree search instruction. In thisimplementation, the free search instruction found in a firewall treesearch is interpreted as a fail instruction.

i) Example of a Chained Lookup

As described above, lookups can be chained to allow for a combination offiltering and lookup operations to be invoked for each packet. Anexample of a chained lookup is: Index Table Lookup->FirewallFilter->Jtree Lookup->Firewall Filter. This sequence corresponds toindex table lookup that is used to index a longest prefix match routelookup (jtree) with firewall filtering applied both before and after thejtree lookup. As described above, the data structures stored in memoryin the lookup engine 110 include a table of “starting next hops”. Thestarting next hop can be used to point to the first lookup operation(e.g., the index table lookup). “Intermediate Next Hops” are used topoint to the subsequent lookup operations (to the first firewall filterprogram, one or more Jtrees for longest match prefix route lookup andone or more different firewall filter programs). The intermediate nexthops are the result returned from each intermediate lookup operation.

The example above begins with a starting next hop of index table lookuptype. That is, the index engine 906 c is invoked to perform a lookup inan indicated index table 916. The index step points to an intermediatenext hop that chooses an input firewall filter for the packet. In thisexample, the firewall filter could be selected based on the “incominginterface index”. That is, the pointer offset field in the starting nexthop moves the current key pointer to the incoming interface index fieldof the key. This index is used to look up the table in memory pointingto the different firewall filter programs.

The firewall filter programs are stored and associated with firewallengine 906 a. The selected firewall filter program is executed by thefirewall engine 906 a just like a processor executing cpu instructions.Some firewall filter instructions refer to data in the key buffer toperform match operations. Depending on the contents of the packet, theprogram may decide to filter (drop) the packet and terminate the lookup,or it may exit and return an intermediate next hop (in this example ofjtree lookup type) that updates the current key pointer (in this exampleto the “IP Destination Address” field of the key).

In this example, the third lookup operation is a jtree lookup. The treesearch engine 906 d performs a jtree lookup on the IP destinationaddress in the key buffer to find the route with the longest matchingprefix. The result is an extended next hop which contains instructionsto place the final destination for the packet based on the jtree lookupresult in the result buffer. This is the destination the packet shouldbe forwarded to if the packet does not get dropped by the firewallfilter in the final lookup step. The next hop returned as a result fromthis jtree lookup operation also contains an intermediate extended nexthop of firewall filter type which points to a firewall program. Theprogram may decide to filter (drop) the packet in which case the lookupwill terminate and the packet will be dropped. If the program decides tokeep the packet, the filter program will simply terminate without takingany special action. In this case, the current contents of the resultbuffer, which is the result of the jtree lookup in the previous step,will be used to forward the packet to its destination appropriately.

Output Switch

Referring now to FIG. 11, output switch 102 includes a controllerinterface 1505, one or more memory inputs 1502 (1502-0 through 1502-7,one for each memory bank), one or more outputs 1504 (1504-0 through1504-7, one for each multi-function multiport), a result processor 1506and an output processor 1508. Output switch 102 performs four functions:receive output results, process output results, receive cells frommemory and output cells to output ports.

a) Transfers from Memory to the Output Switch

Cells from memory are received at memory inputs 1502 and transferred tooutput processor 1508. Cells are transferred based on read requestsreceived at the input switch from multi-function multiports. Each cellincludes an output port identifier and cell data.

Output processor 1508 decodes the destination multi-function multiportfrom the cell information received from memory and transfers the celldata to the appropriate outputs 1502. At each cell slot, output switch102 may receive a cell for processing from each bank in global databuffer 104.

b) Transfers from the Output Switch to the Multi-Function Multiports

Output switch 102 receives notification from controller 106 oncontroller interface 1505. Result processor 1506 decodes the result(route) and determines which multi-function multiport(s) 150 is (are) toreceive the route data. Based on the mask in the notification, resultprocessor 1506 transfers the notification to output processor 1508 fortransfer to each multi-function multiport 150 indicated. At each cellslot, output processor 1508 provides (via outputs 1504) a notificationcell to each multi-function multiport 150.

A notification cell includes a header and data field. The headerincludes memory bank source information and route information. Thememory bank source information includes a source identifier forindicating which memory bank provided the cell in data field. The routeinformation contains data from the notification including a next hopindex, packet length, full address and offsets.

Output Section of a Multi-Function Multiport

Referring now to FIGS. 3 and 12 each multi-function multiport 150includes an output switch interface 316, an input switch interface 304including read request queues 305, head and tail queue buffer 318, anoutput request processor 306, an line output interface 308, storagedevice (memory) 310, stream output buffers 312 and output formatter 314.

a) Notification Queues

A multi-function multiport 150 receives notification that a packet is tobe processed in the form of a notification cell received at the outputswitch interface 316.

Output request processor 306 processes notifications, storing each in anappropriate location in head and tail queue buffer 318 and servicingnotification requests as they make their way through the variouspriority queues in head and tail queue buffer 318. The servicing ofrequests results in the generation of a read request to input switch 100associated with the first address in memory where the packet (associatedwith the particular notification) is stored.

Referring now to FIG. 13, head and tail queue buffer 319 includes aplurality of notification queues Q 1700, where Q=4*s, and where s is thenumber of active streams in the multi-function multiport. Unlike theglobal data buffer, the queues Q are implemented on a per-port basis.The queues in a respective multi-function multiport store only thosenotifications associated with streams to be outputted from therespective port of the multi-function multiport. Each queue is itselfdivided into a head region 1702, a tail region 1704 and a body region1706. The head and tail region for a queue are stored in the head andtail queue buffer 318.

The size of the portion of the head and tail queue buffer dedicated toeach stream is fixed at initialization time and is proportional to thepeak bandwidth of its stream. The partitions between portions are “hard”in the sense that a stream cannot use more than the memory allocated toit. The partitions between queues associated with the same stream are“soft”. The size of an individual queue is proportional to the nominalbandwidth allocated to its queue. The body region of the notificationqueue is stored in the notification area 319 (FIG. 3) of the memorysection 290 of the multi-function multiport 150. Each stream is assigned4 queues (the body portions of the priority queues) in the notificationarea 319 (FIG. 3). The body region can be sized to be ⅕ of the overallmemory section.

Each queue associated with a given stream is serviced according to apriority scheme. Notifications that are received by the output requestprocessor 306 are loaded into an appropriate queue associated with astream based on the priority of the notification. Priority fornotifications can be set by an external source and may be included inthe packet received by the router. Alternatively, controller 106 (FIG.3A) may set the priority depending on the amount of time required toperform the route lookup or other criteria.

Once a queue 1700 has been identified based on the priority informationand stream ID, the output request processor 306 loads the notificationinto the appropriate tail queue 1704. Notifications are transferredbetween the respective head, tail and body portions of a queue based onavailable space by a queue manager (not shown). In one implementation,each notification is 16 bytes, and the notification area 319 is sized tohold 64 bytes. Accordingly, for reasons of bandwidth efficiency, allreads and writes to the notification area are done using 64-byte cellscontaining four 16-byte notifications each. The head and tail of eachqueue is sized to store only a small number of notifications, the bulkof queue storage being provided by the notification area in themulti-function multiport memory bank. As long as space is availableon-chip (on the multiport) to hold the notifications for a queue, thenotification area is completely bypassed. When on-chip space runs out,the notification area acts as the large “middle” of the queue, with afew notifications at the head and tail being held on-chip.

While the size of the notification area will tend to limit the numbersof dropped packets, occasionally a queue will become full. Outputrequest processor includes a drop engine (not shown) for determiningwhich entries in a particular queue are to be dropped based on apredefined algorithm. In one implementation, the drop engine institutesa programmable random early drop routine. The routine is programmable inthat the user can define one or more parameters, random in that a randomnumber generator is used to determine whether a entry will be dropped.Early refers dropping from the head of the queue.

The programmable random early drop routine may be implemented insoftware and when executed performs the following sequence ofoperations. The process begins by calculating the amount of data storedin a particular queue. This information is stored in the form of afraction (or percentage) of fullness. Thereafter, a drop criterion isdetermined based on the fraction of fullness. In one implementation, atable of drop criterion values ranging from zero to one is mappedagainst fullness fractional values. The drop engine then derives arandom number from zero to one. The random number may be generated by arandom number generator or other means as is known in the art. Acomparison is made between the random number generated and the dropcriterion value. Thereafter, the entry at the head of the particularqueue is dropped if the random number generated is larger than the dropcriterion. Alternatively, the drop engine could avoid the drop if therandom number generated is less than the drop criterion. The drop engineoperates on each queue at a regular interval to assure that the queuesdo not overflow and a orderly method of dropping packets is achieved ifrequired. This process is extremely helpful when transmitting packetsacross the Internet.

b) Per Bank Notification Queues

Each stream includes four queues 1700 that are serviced using a weightedround robin discipline. The weighting is used to reflect the priorityassociated with a given queue. For example, the four queues for a givenstream may be serviced in the following ratios: Q1 at 50%, Q2 at 25%, Q3at 15% and Q4 at 10%.

The multi-function multiport maintains four cell pointers for eachqueue: start, end, head, and tail. In one implementation, each pointeris 23 bits long and can address the entire memory associated with themulti-function multiport. The start and end pointers mark the boundariesof the queue's region, while the head and tail pointers point to thenext cell (notification) to read and next cell to write respectively.The head and tail pointers are restricted to align within the regiondefined by the start and end pointers, and standard wraparoundarithmetic is performed when incrementing these pointers.

Given the description above, it should be clear that the region for aqueue can be as small as one cell and as large as the entire memorybank. It is up to the software to configure the pointers atinitialization time to define the sizes of the regions, and to ensurethat regions are non-overlapping with each other and with the memoryallocated to the global packet buffer.

Typically, the software is used to allocate memory to a streamproportional to the stream's bandwidth.

c) Read Request Generation

Output request processor 306 services the queues to extractnotifications from the head regions of queues 1700. Output requestprocessor generates a first read request based on the full addressreceived from output switch 102. Thereafter subsequent read requests aregenerated for transmission to the input switch based on the offsetinformation provided in the request (in the notification cell) orindirect cells (as will be described below). Read requests include astream identifier and a full address. Read requests are sent by theoutput request processor to an appropriate read request queue 305. Oneread request queue 305 is provided for each bank of memory.

In one implementation, if the packet length, as determined from theroute information provided with the notification cell, is greater thanfive (5) cells, then the output request processor first requests thetransfer (read from memory) of the first indirect cell associated withthe packet. This is accomplished by computing the address of theindirect cell based on the full address and the offsets provided in thenotification cell. After the indirect cell request is generated, theoutput request processor generates read requests for the remaining cellsin the packet based on the full address and the offsets provided in thenotification cell. Upon receipt of a indirect cell from the outputswitch 102, output request processor continues to generate read requestsfor the remaining cells in the packet based on the offset informationcontained within the indirect cell.

Subsequent indirect cells are retrieved in a similar fashion. That is,at the time for reading the next indirect cell, the address of the nextindirect cell is computed based on the last offset stored in theprevious indirect cell. The timing of retrieving the indirect cells isaccomplished such that no delays in the output stream are incurred. Eachsubsequent indirect cell is retrieved prior to the end of the processingof the prior indirect cell. In this way, once the output stream isinitialized, no buffering of data is required and no interruptions dueto the latency associated with the retrieval process are experienced.

Output requests to an individual memory bank are processed strictly inorder. That is, the multi-function multiport may track each requestissued to a memory bank (through the read request queues) and is assuredthat the data received in response to a series of requests to the samememory bank will be strictly delivered according to the sequence orpattern in which they were issued. Output request processor 306 keepstrack of requests generated for each memory bank through the use ofreply queues (not shown). The request queue contains a stream number anda read address. When a request is issued to memory, the entry is removedfrom the request queue and the stream number portion is placed in anassociated reply queue. When a reply is received, the entry at the headof the reply queue is removed and the reply is sent to the stream number(in stream output buffer 312) indicated by the stream number retrievedfrom the reply queue.

As cells are received back at the multi-function multiport 150(responsive to the read requests), they are stored in an associatedstream output buffer 312. Stream output buffer 312 includes a pluralityof FIFOs, one for each stream. Each cell received for a stream is placedin the streams associated FIFO. For given packet, the multi-functionmultiport stores a fixed number of cells (in the FIFO) required toprovide a streamed output prior to initializing the output of the streamto line output interface 308. In one implementation of the presentinvention, twelve cells are stored prior to beginning output (streamdata) from the output port. The selection of the number of cells forstorage in output buffer 312 is based on the latency in the read process(number of clock cycles between a read request from an multi-functionmultiport and the arrival of the cell associated with the read requestto the output port).

Output formatter 314 receives the cells from output buffer 312 andcouples the data with media header information stored in memory 310.Each request (notification) received from output switch 102 includes anext hop index. The next hop index indicates the starting address inmemory 310 of the media header information associated with a given typeof transmission (derived from the destination of the packet). Mediaheader information stored in memory 310 may be loaded uponinitialization of the router and updated by the controller as required.Output formatter 314 couples the cell data returned from memory with theappropriate media header to generate a proper packet for transfer out ofrouter 20 on the line output interface 308.

Packet Routing Overview

Referring now to FIG. 14, in a method of routing packets through aswitch a packet is received at a multi-function multiport (1800). Themulti-function multiport divides the packet into fixed length cells andtransfers the cells to an input switch (1802). Input switch removes thekey information from the first cell in a packet and stores ittemporarily in a key buffer (1804). Thereafter the input switch routesthe cells to memory banks resident in the multi-function multiports in atime division multiplexed manner (1806). The input switch stores thefirst address in memory where the first cell is stored and computesoffsets for each additional cell associated with the offset in memoryfor the next contiguous memory bank into which the next cell is written(1808). The input switch creates indirect cells to store linkinginformation for the packet if the packet length exceeds five cells(1810). If the number of cells exceeds the number of available offsetsin an indirect cell, then the old indirect cell is stored in memory anda new indirect cell is created and loaded based on the offsetscalculated for each new cell received at the input switch.

When the packet (and its indirect cells if any) have been stored inmemory, then the key, full address of the first cell and offsetinformation is transferred as a lookup request to a controller (1814).The controller performs a lookup operation that can include a pluralityof chained lookup operations and generates a result. The result includesthe destination port (multi-function multiport), address, offsetinformation and next hop index (1816). A notification including theresult is transferred to the output switch for transfer to theappropriate multi-function multiport (1818).

Upon receipt of a notification, the multi-function multiport generatesread requests a cell at a time to the input switch for the dataassociated with the packet (1820). The input switch issues the readrequests in a time division multiplexed fashion generating a singlerequest to each memory bank per cell slot (1822). When the memory bankreceives the request from the input switch, cell data and amulti-function multiport identifier associated with the request aretransferred to the output switch (1824). Again, at each cell slot, theoutput switch transfers a single cell to each of the multi-functionmultiports. Upon receipt, the multi-function multiport couples the celldata with media header information and streams the data to thedestination (1826).

Alternative Implementations

The present invention has been described in terms of specificembodiments, which are illustrative of the invention and not to beconstrued as limiting.

For example, the system can be configured to run the same jtree over twodifferent fields in the key (this is so the system can support a matchwhich is true when one of a list of prefixes matches either thedestination address or the source address in a packet).

Next hops can be designed to support many features in addition to packetforwarding. Additional packet processing features that are possibleinclude but are not limited to accounting, sampling, quality of service,flow policing, and load balancing.

For accounting, fields in an intermediate or extended next hop canindicate that a specific counter should be incremented. For example, theroutes in a jtree for longest prefix match lookup are next hops. Ifthese next hops point to different counters, the counters can be used tocount the number of packets taking each route. Note that more than onenext hop can be set up to point to the same counter. For supportingsampling and quality of service, next hops can contain fields thatmodify sampling enable bits and quality of service values stored in theresult buffer. A lookup step can be added to the chain of lookups totest certain fields in the key to select the next hop to make thedesired modification.

For supporting flow policing, next hops can contain pointers to datastructures storing policing parameters. A firewall filtering step can beinserted into a chain of lookups to select a set of policing parametersbased on different fields in the key buffer.

For supporting load balancing, an intermediate next hop can be designedto point to more than one final next hop for selection.

The data structures (jtrees, index tables, filter programs) stored inmemory 920 (of FIG. 9) to support the various lookup operations can beatomically updated so that updates to the data structures can beperformed at the same time lookup operations are being executed. Each ofthe data structures includes a pointer to a next operation. The pointersare of the form of a next hop. The next hop may be an intermediate nexthop, or a final next hop. Modifications can be made to a chain (a seriesof lookups that are to be performed for a particular type of packet(e.g., incoming interface)) without requiring a current lookup operationto be terminated. That is, an entry in the chain can be removed byupdating the pointer in the lookup specified one entry in the chainabove the item to be removed to point to the entry in the chain that isone entry after the deleted item. The update is atomic, in that existingoperations are not affected, and any subsequent results returned afterthe update will branch to the appropriate next entry in the lookupchain. All changes can be resolved to a single write operation.Similarly, a lookup can be added to the chain by adding the datastructure to memory including designating the result that is returned bythe added entry point to the entry in the chain at the point just afterwhere the new entry is to be included in the chain. Finally, the pointerin the entry (the result returned) just above the point where the newentry in the chain is to be included is updated to point to the newentry. Again, existing operations are not required to be terminated andsubsequent chain processing operations (that are executed after thechange) will include the added lookup.

Similarly, within the data structures, updates to individual portions ofthe branching operations (e.g., branches in a tree, a branch in a filterprogram, or a result returned in an index search) can be performedatomically.

The Internet may become the fundamental communications infrastructurefor public network access, for private networking, and fornext-generation voice and video applications. Implementations consistentwith the principles of the present invention boost packet forwardingperformance to match the speed and capacity of the emerging opticalinfrastructure. Implementations add IP intelligence to raw forwardingperformance. The result is a substantial and predictable increase inpacket processing performance when compared to software-based solutions.Consequently, packet forwarding performance does not have to besacrificed to deploy the IP services that subscribers demand, therebycreating a competitive advantage in a rapidly evolving marketplace.

ASIC-based packet processors consistent with the principles of theinvention, configured using appropriate software, deliver the toolsneeded to scale security management at any bandwidth and at any point inthe network. Not only does the ASIC-based packet processor deliverwire-rate forwarding performance and unprecedented visibility intonetwork operations, it also supports packet filtering, packet samplingand logging, packet counting, and load balancing.

Implementations of the ASIC-based packet processor are first in theindustry to support wire-rate forwarding performance. The ASIC-basedpacket processor retains this performance while adding a rich set offeatures that can be deployed both in the core and at the edges of anetwork.

The ASIC-based packet processor provides breakthrough technology thatallows routers to forward traffic at wire-rate speeds. A benchmark offorwarding 40-byte packets at 40 Mbps with an 80,000-prefix routingtable may be achieved. In addition to wire-rate forwarding, this ASICsupports industrial-strength routing protocol implementations, a routingpolicy definition language, reliable performance under stress, flexiblemultiprotocol label switching (MPLS) traffic engineering, andclass-of-service (CoS) queuing. The ASIC-based packet processor deliversbest-of-class functionality in the network core.

The ASIC-based packet processor delivers 40-Mpps forwarding rate; italso adds the packet processing features needed to build a competitiveadvantage in a rapidly evolving industry. Offering rich packetprocessing, such as filtering, sampling, logging, counting, and loadbalancing, the ASIC-based packet processor maintains high performance.Implementations may be used in Internet backbone routers. Software maybe downloaded to activate the ASIC-based packet processor features onall interfaces, now and in the future.

The ASIC-based packet processor may reside on the packet forwardingengine, which may be cleanly separated from the routing engine. Theseparation of forwarding and routing functions ensures that stressexperienced by one component does not adversely affect the performanceof the other because there is no overlap of required resources. Routingfluctuations and network instability do not limit the forwarding ofpackets. The wire-rate forwarding performance of 40 Mpps ensures thatthe ASIC-based packet processor scales well with large, complexforwarding tables. Predictable performance with advanced features, suchas filtering, is maintained because of the ASIC-based packet processor'sinherent over-sized design with respect to interfaces, as well as thefact that once configured, the routing engine compiles and optimizesfilters before sending them to the ASIC-based packet processor.Additionally, the packet forwarding engine pooled resource designcombines with the over-sized ASIC-based packet processor to ensureadditional performance headroom, even with value-added features enabled.

Given the ASIC-based packet processor may be over-sized relative torealistic routing configurations and realistic traffic (in terms ofinterface utilization and distribution of packet sizes), in practice thepacket load is not expected to exceed the capacity of the ASIC-basedpacket processor. This expectation is particularly true for the smallerrouters where the ASIC-based packet processor may be even moreover-sized. These routers may be used in access roles, where servicesare usually implemented and hence, where the increased amount ofpacket-processing capacity is a competitive advantage.

Designing a set of tests to characterize a router's basic forwardingperformance may be a relatively straightforward task because fewvariables need to be taken into account. However, developing ameaningful set of benchmarks to describe a router's forwardingperformance with filtering enabled may be significantly more complexbecause of the large number of variables involved. Such variables mayinclude the number of router interfaces that are configured with packetfilters, the number of input and output filters installed on eachinterface, the size and complexity of the filters used during the test,the volume of traffic arriving at each interface, the distribution ofpacket sizes on each interface, the aggregate volume of traffic enteringthe system at any time, and the content of the traffic's packet headers.

In the absence of standard benchmark tests, the best approach toquantifying filtering performance is to test the platform under extremeconditions, for example, using fully configured systems with fullInternet routing tables and multiple unique and complex packet filters.The results, as shown in FIG. 15, may demonstrate that the ASIC-basedpacket processor delivers predictably high forwarding performance whilefiltering packets for any traffic profile and any filter set likelyencountered in a production network.

Three of the applications delivered by the ASIC-based packet processorinclude packet filtering, traffic analysis, and load balancing.

Packet Filtering

Packet filtering is the ability to selectively control the flow ofpackets to or from an interface by examining information in eachpacket's header. The ASIC-based packet processor providesperformance-based packet filtering of inbound and outbound traffic basedon any combination of fields such as source and destination IPaddresses, IP protocol (such as TCP, UDP, and ICMP), source anddestination UDP and TCP ports, DiffServ byte, IP fragmentation offsetand control fields, and TCP control bits.

High-performance filtering is accomplished using a flexible method ofprogramming the ASIC-based packet processor. Filters may be written anddownloaded onto the ASIC-based packet processor using an easy-to-usecommand-line interface. A compiler then optimizes and compiles thefilters for the ASIC-based packet processor for quick and efficientpacket processing.

When a packet matches a user-configured filter rule, the router may takeany of a number of actions, such as accept the packet, silently discardthe packet without sending an ICMP message, reject the packet and sendan ICMP message, etc. In addition to these actions, the router may beconfigured to perform processes, such as sampling the packet,incrementing a counter, and logging the packet. For example, a packetcan be configured to be accepted and sampled, to be discarded, counted,and logged, or to be rejected and logged.

Filtering may be useful for a variety of applications, includingprotecting the core and protecting subscriber networks. For example,ASIC-based packet filters may be used to protect the network corefrom-unwanted traffic, source address spoofing, and other types ofunauthorized management access.

Traditionally, two types of tools are used in a layered fashion toprotect core routers. The first line of defense is the router's remoteaccess management policy, which is essentially an IP address list.Management access to the router (for example, using Telnet or SNMP)requires an allowed source IP address. After the source IP address isverified, a second tool, such as passwords or one-time passwords,provides a second layer of security.

The ASIC-based packet processor adds a third layer of security toprotect against attacks to the core. Applying filters that performsource address verification at network ingress points ensures thathackers cannot spoof the source address of network operation center(NOC) systems at the ingress' edge of the network. An exemplaryconfiguration is illustrated in FIG. 16.

Source address verification can also play an important role inprohibiting the launch of denial-of-service (DoS) attacks fromsubscriber sites. Many hackers attempt to obscure true identities byusing a forged IP source address. The filtering, sampling, and loggingcapabilities of the ASIC-based packet processor can be used to trace aDoS attack back to the source without negatively impacting theforwarding performance in the network core.

These robust filters run all the time, examining and, if needed,dropping packets before they enter a network. Never before has it beenpossible to implement this type of network armor using traditionalfilters because the forwarding performance of traditional routers usingsoftware-based filtering degrades to unacceptable levels.

The added benefits of using ASIC-based filters for core routerfortification include an additional layer of security provided toprotect routers from unauthorized access, and permanent filters do notimpact packet forwarding performance. Historically, the burden of packetfiltering has been placed on the subscriber. Filters were configured onsubscriber edge routers (customer premise equipment (CPE)) to examineincoming traffic from the provider side of the subscriber-providercircuit. Network administrators at the subscriber site were heavilytasked with creating and managing the traffic filters.

As increasing numbers of subscribers migrate to high-bandwidth circuits,existing CPE platforms are unable to offer predictable and stableperformance when filtering is enabled. Operationally, it is moreattractive either to manage filtering on the provider side of thecircuit or to deploy equipment that delivers full performance at the CPEfor high-bandwidth subscribers. ASIC-based filtering enables protectionof subscriber networks using outsourced packet filters configured in theprovider edge routers. For example, outsourced packet filters can beconfigured to accept, discard, or reject packets on the egress interfacefacing the subscriber edge. An exemplary configuration is illustrated inFIG. 17.

Outsourced packet filters implemented using the ASIC-based packetprocessor offer several distinct advantages over traditionallyarchitected filters. For example, bandwidth on the provider-subscribercircuit is conserved. Outbound filters can be applied before trafficever leaves the network. These filters eliminate unwanted andunnecessary traffic from ever wasting bandwidth on theprovider-subscriber circuit. The subscriber does not have to deal withthe complexity of creating and managing filters. Forwarding performanceon the edge router is not negatively impacted. The structured,database-like configuration tools supported in commercially availablesoftware significantly ease the burden of managing customer filteringconfigurations at the edge router. Outsourced packet filtering servicescan be offered as a value-added, revenue-generating differentiator.

Packet filtering can be used on an aggregation router to prevent sourceaddress spoofing from subscriber subnets. FIG. 18 shows a typicalaggregation router that provides access for Subscriber 2 across anOC-3/STM-1 circuit assigned to interface so-1/1/1. Unlike a classicfirewall filter, which rejects all traffic that is not explicitlyaccepted by a filter term, the default action for this filter is toaccept all traffic that is not explicitly rejected. A key assumption inthis example is that the subscriber is single-homed. Otherwise, sourceaddress verification may not be possible on a single circuit withoutknowledge of the source addresses provided by other service providers.

The filter installed on the ingress interface of the service provider'sedge router verifies that the source address of each subscriber packetis within the prefix range assigned to the subscriber. By logging andcounting each packet with an invalid source address, a spoofing problemand its magnitude can quickly be identified. The pseudo code for thisexample is as follows:

IF (source address < >subscriber's prefixes)

THEN

-   -   COUNT PACKET sub-2-unauth-src-addr    -   LOG PACKET    -   DISCARD PACKET

ENDIF

The actual filter consists of two terms. The first term,sub-2-prefix-check, examines each packet for an unauthorized sourceaddress; if there is a match, the packet is silently discarded, logged,and counted. The second term, default-action, has no match statements,so it matches all packets that fall through and are processed by thisterm. The default action for this filter is to accept all packets thatwere not explicitly rejected by previous filter terms.

Assuming that the subscriber's source address prefix is 135.6/16, thesoftware syntax to implement the filter is as follows:

[edit firewall] filter subscriber-2-ingress { term sub-2-prefix-check {from { source-address { 0.0.0.0/0; 135.6/16 except; } } then { countsub-2-unauth-src-addr; log; discard; } } term default-action { thenaccept; } }

To assign this filter to interface so-1/1/1, the software syntax is asfollows:

[edit interface] interfaces so-1/1/1 {. unit 0 { family inet { filter {input subscriber-2-ingress; } } } }DOS Attack Packet Filtering and Tracing

The ASIC-based packet processor provides a powerful set of tools thatallows filtering and tracing of DoS attacks over high-speed circuitswhile still maintaining predictable router forwarding performance.Packet filtering is an effective means of countering smurf and other DoSattacks by blocking intrusive packets before they reach their victim.Log filters enable one to trace the origin of a DoS attack across aprovider's network to the assault's ingress point.

This example demonstrates how to use packet filtering and logging totrace a smurf-based DoS attack. FIG. 19 shows how a hacker launching asmurf attack by sending an ICMP echo request (ping) packet with aspoofed source address. The source address that the hacker uses in theICMP echo request packet is the address of the target of the attack. TheICMP echo request packet is transmitted to the directed broadcastaddress of the proxy network. When hosts on the proxy network receivethe hacker's ICMP echo request, they respond by transmitting ICMP echoresponse messages to the victim. Using the directed broadcast addressallows the hacker to amplify the number of ICMP echo responses that thevictim receives. Using multiple proxy networks, the hacker can furtherintensify this attack.

The pseudo code to discard and log all packets belonging to a smurfattack is as follows:

IF (destination address=victim) AND

-   -   protocol address=victim) AND    -   (ICMP-type=echo response)

THEN

-   -   LOG PACKET    -   DISCARD PACKET

ENDIF

The software filter for this example contains two terms. The first termlogs and then discards ICMP echo response packets, while the second termexecutes the filter's default action and accepts all other packets. Asshown in FIG. 20, the address of the victim is 200.7.5.3 and theprovider edge router interface toward the victim's network is so-4/0/0.

[edit firewall] filter dos-tracer { term dos-discard-log { from {destination-address 200.7.5.3; protocol icmp; icmp-type echo response; }then { log; discard; } } term default-action { then accept; } }

The following software syntax applies the filter to the provider edgerouter's outbound interface toward the victim's network.

[edit interfaces] interfaces so-4/0/0 { unit 0 { family inet { filter {output dos-tracer; } } } }

A sample log generated by the filter is as follows:

user@host> show firewall log Time doc-log Interface Pro Src address Destaddress 03:09:07 - A so-4/3/0 ICM 193.5.1.2 200.7.5.3 03:09:07 - Aso-4/3/0 ICM 193.5.1.3 200.7.5.3 03:09:07 - A so-4/3/0 ICM 193.5.1.7200.7.5.3 03:09:08 - A so-4/3/0 ICM 193.5.1.8 200.7.5.3

The log lists the following data for each logged packet: the time thepacket was received; the router interface on which the logged packet isreceived (so-4/3/0); the protocol (ICMP); the source address (a host onthe proxy network), and the destination address (200.7.5.3, the victim).Once it is known that attack packets arrive on interface so-4/3/0 of theprovider edge router, the upstream neighbor can easily be identified andthe software filter installed on the outbound interface of the previoushop router. By repeating this process hop by hop, the attack can betraced back to the point that it enters the network.

Fortifying Core Routers Against NOC Spoofing Attacks

The ASIC-based packet filtering features can be used to increase thesecurity of service provider core routers by preventing hackers fromspoofing NOC addresses. In this example, a filter discards packetsarriving on the inbound circuit at a peering point if they contain aspoofed NOC source address. A packet received on this interface with aNOC prefix indicates that an attack is currently underway using aspoofed NOC address.

FIG. 21 shows a typical service provider with a peering point to otherservice providers on the Internet. The pseudo code and the softwarefilter syntax for this example are as follows:

IF (source-address = NOC) THEN LOG PACKET DISCARD PACKET ENDIF [editfirewall] filter peer-input { term chk-NOC-address { from {source-address 221.5.6/24; } then { log; discard; } } termdefault-action { then accept; } }

This filter complements, rather than replaces, security measures on eachrouter, including remote management access control and strongauthentication. The filter makes it significantly more difficult forhackers to break into routers and tamper with configuration parameters.Note that this example assumes subscriber source address verification isenabled on all aggregation routers as described in the first example.Subscriber verification filters prevent NOC source address spoofingattacks from entering the network at subscriber ingress points.

Fortifying the Routing Engine with Packet Filtering

This example illustrates how the packet filtering features of theASIC-based packet processor can be used to protect the routing engine ofa router. The routing engine can be protected by applying packet filtersto either a physical interface or to the router's loopback interface.

Example: Router Physical Interface

ASIC-based filtering can eliminate unnecessary processing by the routingengine, while still maintaining predictable router forwardingperformance. In FIG. 22, Service Provider 1 wants to peer only withService Provider 4, but not Service Provider 2 or Service Provider 3.Since the filter operates on an interface to an exchange point, thedefault action for the filter is to accept all packets that are notexplicitly rejected by prevous filter terms. The pseudo code andsoftware syntax to implement this filter are as follows:

IF (source address < > valid peer) THEN COUNT PACKET illegal-bgp-peersDISCARD PACKET ENDIF [edit firewall] fitter peer-input { . . . termillicit-bgp-peers; from { source-address { 0.0.0.0/0; 194.1.2.4/32except; } protocol tcp; destination-port bgp; } then { countillegal-bgp-peers; discard; } } . . . term default-action { then accept;} }

The following software syntax applies the peer-input filter to therouter's inbound Gigabit Ethernet interface attached to the exchangeLAN:

[edit interfaces] interfaces ge-1/2/0 { unit 0 { family inet { filter {input peer-input; } } } }

As with the NOC example, packet filtering complements, rather thansubstitutes for prudent security measures. For BGP, such measures wouldinclude using MDS to authenticate each peer and BGP group that acceptsonly specified BGP systems as peers.

Example: Router Loopback Interface

ASIC-based packet filtering can be used to restrict the packets that canpass from any of the router's physical interfaces to the routing engine.Such filters are useful in protecting the IP services that run on therouting engine, such as Telnet, SSH, or BGP, from DoS attacks. Insteadof assigning these filters to a specific physical interface, they can beassigned to the router's loopback interface, lo0, which is the interfaceto the routing engine, and as such, carries no transit traffic. As shownin FIG. 23, if a filter is applied to the lo0 interface, the filteraffects only those packets received or transmitted by the routingengine.

For example, to discard Telnet and SSH access to all but the192.168.1/24 subnet on all interfaces, the following software syntax canbe used. The terminal-access term also logs unauthorized Telnet or SSHtraffic to the loopback interface's log buffer.

[edit firewall] filter routing-engine-input { . . . term terminal-access{ from { source-address { 0.0.0.0/0 192.168.1/24 except; } protocol tcp;destination-port [ssh | telnet]; } then { log; discard; } }

The following software syntax applies the routing-engine-input filter tothe router's loopback interface:

[edit interfaces] interfaces lo0 { unit 0 { family inet { filter { inputrouting-engine-input; } } } }

Traffic Analysis

As network bandwidth becomes ever higher and networks rapidly grow,scalable data capture and analysis tools are needed to allow visibilityinto the operations of the networks. Traffic analysis tools used inconjunction with packet filters enable such visibility. The filteridentifies specific packets to be examined, and the traffic analysistools sample, log, and count the selected packets.

Traffic analysis tools allow traffic to be examined, and networks to beplanned and designed, for example, in understanding the nature of thenetwork traffic. Traffic can be characterized in a number of dimensions,including distribution of packet sizes, transport protocols, popularapplications (such as the Web voice and video) tunneled protocols and IPmulticast. Future capacity network design and deployment can be plannedfor both internal circuits (intradomain) and external circuits(interdomain) as well as determining whether new peering relationshipsneed to be established. Future equipment requirements can be identifiedby analyzing the network performance to determine necessary forwardingrates for prevalent packets of a specific size. Profiles can bedeveloped that define normal network operations and a plan of action canbe developed, for instance, when the operation of the network is outsideof the defined boundaries.

Packet sampling and logging allow the router to examine packets as theytraverse the system. One of the inherent problems in packet sampling isthat it is orthogonal to the system's main function: processing thegreatest number of packets as quickly as possible. A major benefit ofrouting architecture in which the CPU is not directly involved in theprocessing of transit traffic, is that the router remains lightlyutilized and is available for storing packet headers and performingonline analysis for local display.

ASIC-based packet processors use statistical sampling that examines auser-configurable percentage of the traffic traversing the system.Sampling theory shows that statistical sampling can be quite accurate ifthe sampling parameters are properly selected. When a packet matches thefiltering conditions, the router marks it as a candidate for sampling bysetting a bit in the packet notification. Typically, the router sets thesample bit if the packet matches a filter. However, having the samplebit set does not mean that the packet notification is sampled, only thatthe specific packet notification is a candidate for sampling.

The packet filter on the configured interface sets the sample bit on allthe sampled packets. For each packet that has the sample bit set, theASIC-based packet processor generates a random number. If the randomnumber is less than a user-defined threshold, the packet is sampled. Ifa packet is sampled, the router can sample the next user-specifiednumber of packets without running the random number generator.

When a packet is sampled, the router writes the packet headers to a fileon the hard disk of the routing engine. The information may includesource and destination IP addresses, IP protocol, source and destinationport, packet length, DiffServ byte, IP fragment flags, and TCP flags.

Logging is similar to sampling. However, logging can be used for instantnotification of ongoing network events. Logging examines every packetthat matches the filter and displays the matches in realtime on theconsole. The router does not log data on the hard disk; the logged datacan be accessed by using the routers command line interface.

Packet Sampling

The ASIC-based packet processor allows sampling of IP traffic based onparticular input interfaces or various fields in the packet header.Traffic sampling can be used to monitor all logical interfaces, specificlogical interfaces, specific protocols on one or more interfaces, arange of addresses on a logical interface, or individual IP addresses.Information about the sampled packets is saved to files on the router'shard disk drive.

Using packet sampling filters, a representative sampling of packetheaders from those packets that match a filter's conditions can beeasily captured and stored. A randomized approach is a more practicalsolution than attempting to sample each and every packet (as executed intraditional routers) given the amount of data flowing through an OC192c/STM-64 circuit. Randomization determines whether a particularpacket is sampling based on a user configurable sampling rate. Thesampling rate is defined as one divided by a user-specified integer.This user specified sample rate is compared to a random number generatedby the ASIC-based packet processor. If the resulting value is below thespecified rate, then the packet is sampled. Over the course of manysamples the sampling rate equals the rate specified in the userconfiguration. Our implementation eliminates the bias from sampling justevery nth packet, and thus makes the data more statistically meaningful.

All packets that match the conditions of a sampling filter are marked ascandidates for sampling. By configuring a user specified run length ofup to 20 packets, marked packets can be sampled that are adjacent to thepacket currently being sampled. For example, if a sampling rate of 1,000is configured to trigger the sampling of 1 out of every 1,000 markedpackets and also the run-length is set to a value of 3, the softwarealso samples the next two packets that are marked for sampling. Bydefault, five adjacent packets are sampled after the initial triggerevent.

Below is the output of a typical sample file containing captured headerinformation:

user<host> monitor start /var/log/sample•file # Apr 5 20:31:21 #    DestSrc Dest Src Proto TOS Pkt Intf IP  TCP #    addr addr port port len numfrag flags 211.211.221.1 10.2.0.25 60 39 6 0x0 40 9 0x0 0x10211.211.221.1 10.2.0.25 66 1024 6 0x0 60 9 0x4000 0x2 211.211.221.110.2.0.25 80 39 6 0x0 10 9 0x0 0x10 211.211.221.1 10.2.0.25 56 39 6 0x040 9 0x0 0x2

A typical application for packet sampling is to determine the ultimatedestination, volume, and content of traffic flowing between serviceprovider networks. This information can be used to decide if additionalcapacity needs to be provisioned or if new peering relationships need tobe established.

FIG. 24 shows a topology where Service Provider 1 uses packet samplingto determine if it should establish a direct peering relationship withService Provider 3, rather than relying on Service Provider 2 to furnishtransit service.

For this example, the filter provider-1-out is applied to the circuitbetween Service Provider 1 and Service Provider 2. The filter termsample-all-to-provider-2 samples all packets forwarded on this circuit.The syntax to define the filter and assign it to interface so/2/2/0 isas follows:

[edit firewall] filter provider-1-out { . . . termsample-all-to-provider-2 { from { destination-address 0.0.0.0/0; } then{ sample; accept; } } . . . term default-action then accept; } } [editinterfaces] interfaces so-2/2/0 { unit 0 { family inet { filter { outputprovider-1-out; } } } }

To specify a sampling rate of 1 in 1,000 packets and to direct thesampling output to the file provider-two-out.txt, the following syntaxmay be used:

[edit forwarding-options] sampling input { family met rate 1000; rate1000; } } output file filename provider-two-out.txt; } }

After the sampled data is collected, an off-line analysis toot can beused to examine the sample file and determine whether the volume oftraffic flowing to Service Provider 3 merits a direct peeringrelationship between Service Provider 1 and Service Provider 3.

Sampling Example

By combining this sampling capability with filtering, strategic trafficanalysis can be performed. Example analyses include determining whetheradding a direct peering circuit would improve efficiency and responsetime, and determining how much traffic from a specific source address isreaching a hosting site. An exemplary configuration is illustrated inFIG. 25.

Packet Counting

Packet counting operates in real time and at the highest priority.Packet counters are 100% accurate even when they run over OC-48c/STM-16or OC192c/STM-64 interfaces. Even at these speeds, filters and counterscan be customized to provide the information to effectively operate,maintain, and plan a network.

Counters run in the ASIC-based packet processor at wire speed andcompletely independent, from the CPU. The ASIC-based packet processorcan be configured to run full time to track particular traffic types orto run part time to explore or track the state of the network.

Counting Example

Filters can be configured to target the exact nature of traffic on anetwork and, by counting different packets, provide visibility into thepacket types that are traversing the system. For example, a filter canbe configured that counts all packets sourced from a range ofspecific/24 IP address prefixes entering a network by way of a peeringconnection.

Real-Time Packet Counters

Packet counting provides a very powerful tool for characterizing trafficon service provider networks. Packet counters can be used to performcapacity planning and identify bottlenecks in the core of a network;determine the types of packets transiting or destined to the core andthe relative ratio of one packet type to another (for example, the ratioof mail versus FTP packets); and analyze attacks on service provider orsubscriber networks by counting packets for commonly used attacks (e.g.,ICMP request response packets used in a smurf attack).

The ASIC-based packet processor supports realtime packet counters on allinterfaces at speeds up to OC-192c/STM-64. The match conditions in asoftware filter term are used to determine which packets are counted.These terms may be applied to a single circuit or to a group of circuitsand to either the input and/or output packet stream. Moreover, becausethese filters are implemented in an ASIC, permanent filters may beconfigured to gather historical data, identify traffic trends, determinethe nature of an attack, or to analyze traffic without significantlyimpacting router forwarding performance.

FIG. 26 shows a typical application where Service Provider 3 counts thenumber of HTTP packets that are received from Service Provider 2 and areaddressed to the Web server (215.1.5.23). The pseudo code to implementthis counter is as follows:

IF (destination address=Web Server) AND

-   -   (protocol TCP) AND    -   (destination port=HTTP)

THEN

-   -   COUNT http-to-web-server    -   ACCEPT

ENDIF

The following syntax defines a filter term that implements the packetcounting application specified by the pseudo code:

[edit firewall] filter traffic-from-provider-2 { . . . termhttp-to-web-server from { source-address { 0.0.0.0/0; }destination-address { 215.1.5.23; } protocol tcp; destination-port http;} then { count http-to-web-server; accept; } } . . . term default-term {then accept; } }

The filter is assigned to Interface so 3/0/0 using the following commandsyntax:

[edit interfaces] interfaces so-3/0/0 { unit 0 { family inet { filter {input traffic-from-provider-2; } } } }

A data collection program could be written to periodically access andarchive the counter http-to-web-server. This data is useful forhistorical or trend analysis and to provide an alamn if a specificthreshold value is exceeded over a period of time.

Real-Time Packet Logging

The ASIC-based packet processor allows logging to be performed in realtime and at wire-rate speeds up to OC-192c/STM-64. At times, using thislogging capability might be more appropriate than just counting packets.For instance, logging packets that meet certain criteria is a verypowerful diagnostic tool for characterizing packet traffic in a serviceprovider core network.

The ASIC-based packet processor can be configured with software to logpackets that satisfy the match conditions specified in a filter term.The log is implemented as a buffer that can be viewed using the softwarecommand-line interface. The log entry for a packet includes the time itwas logged, the input circuit, the protocol type, the source address,and the destination address.

This example illustrates how the logging and counting capabilities canbe used of the ASIC-based packet processor to support the creation oftools that permit quick and accurate determination of the magnitude of awide range of network problems. Specifically, FIG. 27 shows how a packetfilter can be used to identify traffic containing an unauthorized sourceaddress on the input circuit from Subscriber 2. The packets that matchthis filter term are counted, logged, and then silently discarded.

IF (source address < >subscriber's prefix)

THEN

-   -   COUNT unauth-src-addr-sub-2    -   LOG    -   DISCARD

ENDIF

Assuming that the subscriber's source address prefix is 135.6/16, thesoftware syntax to implement the log filter is as follows:

[edit firewall] filter subscribor-2-ingress term sub-2-prefix-check fromsource-address 0.0.0.0/0; 135.6/16 except; } } then { countunauth-src-addr-sub-2 log; discard; } } term default-action { thenaccept; } }

To assign this filter to interface so-1/1/1, the software syntax is asfollows:

[edit interfaces] interfaces so-1/1/1 { unit 0 { family inet { filterinput subscriber- 2-ingress; } } } }

A sample log generated by the execution of the subscriber-2-ingressfilter is as follows:

user@host> show firewall log Time doc-log Interface Pro Src address Destaddress 03:09:07 - A so-1/1/1 ICM 193.5.2.2 200.7.5.3 03:09:07 - Aso-1/1/1 ICM 193.5.2.3 200.7.5.3 03:09:07 - A so-1/1/1 ICM 193.5.2.7200.7.5.3 03:09:08 - A so-1/1/1 ICM 193.5.2.8 200.7.5.3

These specific packets are logged because their source addresses are notincluded in the authorized prefix for the subscriber's network(135.6/16). This situation could indicate that some type of sourceaddress spoofing is occurring. Since the protocol is ICMP, the abovepackets could possibly be ICMP echo responses that are part of a smurfattack.

Load Balancing

On a router with parallel circuits, packets are evenly distributedacross the circuits on a per-packet basis. The ASIC-based packetprocessor allows flows to be streamed onto each of the circuits so thatpackets that contain a particular source-destination/port-address pairare forwarded out the same physical interface. In other words, allpackets from a particular flow are transmitted over the same circuit.

Load balancing has two primary benefits: providing extra bandwidth overparallel circuits or equal-cost paths, and reserving packet ordering sothat user TCP sessions operate at peak efficiency. Maintaining packetorder ensures that TCP implementations that support fast retransmit andfast recovery do not slow down because of packet misordering.

The ASIC-based packet processor manipulates the information contained inthe packet header, assigning each packet hash value. This manipulationguarantees that all traffic with the same hash value is transmitted outthe same interface and ensures that the packet order within a TCP flowis maintained.

Load Balancing Example

Load balancing is particularly useful for at least two topologies. Loadbalancing can be used across parallel, point-to-point circuits towards adestination. In this topology, load balancing is effective especiallywhen moving up to the next step in the bandwidth hierarchy cannot bejustified. Load balancing can be used across three equal-cost, multihoppaths. Enabling load balancing allows the system to pick from a numberof equal-cost paths and select the same path for the traffic belongingto each TCP session. An exemplary configuration is illustrated in FIG.28.

Rate Limiting

Sustained congestion is a consequence of under-provisioned,statistically-multiplexed networks. When an access circuit is firstprovisioned, the subscriber and service provider agree on the averagerate (and burstiness) of data transmission over the link. As long as thesubscnber fulfills their part of the contract and transmits packetsaccording to the service level agreement (SLA), the service providerattempts to deliver them in a timely manner. While timely delivery isnot a requirement for traditional data applications, it is critical foremerging, real-time applications, such as VoIP and desktop conferencingthat are intolerant of delay and jitter resulting from networkcongestion.

To support a diverse set of SLAs across all subscribers, it is criticalthat a service provider regulate traffic flows to protect the sharedresources in the core of its network and ensure that each subscriberdoes not consume more than its fair share of bandwidth. To do this,service providers need tools that allow them to determine whether eachsubscriber is honoring their SLA and what actions should be taken if asubscriber attempts to inject out-of-profile traffic into the network.

There are two fundamental approaches to protecting the shared resourcesin the core of a service provider network: traffic shaping and trafficpolicing. Traffic shaping seeks to reduce the potential for networkcongestion by placing packets in a queue with a shaper at the head ofthe queue. These tools smooth out packet flows and regulate the rate andvolume of traffic admitted to the network. There are two fundamentaltraffic-shaping, rate-limiting tools. A traffic-smoothing tooleliminates bursts and presents a steady stream of traffic to thenetwork. This rate-limiting function is commonly implemented using aleaky bucket algorithm. A long-term average traffic rate-shaping toolpermits bursts of a predetermined size and presents a burst-regulatedstream of traffic to the network. This rate-limiting function iscommonly implemented using a token bucket algorithm. Each of these toolssupports different rate-limiting capabilities and results in outputstreams with different characteristics.

Traffic-Smoothing Tool Using a Leaky Bucket Algorithm

The leaky bucket rate-limiting algorithm turns a bursty stream ofpackets into a regular stream of equally spaced packets. FIG. 29illustrates how the leaky bucket rate limiting algorithm operates. Withthe leaky bucket rate-limiting algorithm, an unregulated stream ofpackets is placed into a packet queue controlled by a leaky bucket queueregulator. If the flow presents more packets than the queue can store,the extra packets are discarded. When packets reach the head of thequeue, they are forwarded into the network at a constant rate determinedby the configuration of the queue regulator.

Leaky bucket rate-limiting tools can be used to manage the flow of dataso that packets are not forwarded into the network at a rate greaterthan the network can or is willing to absorb. The length (or depth) ofthe packet queue bounds the amount of delay that a packet can incur atthis traffic shaper in the end-to-end path. However, a packet can incuradditional delay at downstream hops if the remainder of the network isnot adequately provisioned to support the offered load.

Burst-Shaping Tool Using a Token Bucket Algorithm

The token bucket rate-limiting algorithm enforces a long-term averagetransmission rate while permitting bounded bursts. In this approach, atoken bucket is used to manage the queue regulator that controls therate of packet flow into the network. FIG. 30 illustrates how the tokenbucket rate-limiting algorithm operates.

A token generator constantly produces tokens at a rate of R tokens persecond and places them into a token bucket with a depth of D tokens.Assuming that each token grants the ability to transmit a fixed numberof bytes, if the token bucket fills, newly generated tokens arediscarded. At the same time, an unregulated stream of packets arrive andare placed into a packet queue that has a maximum length of L. If theflow delivers more packets than the queue can store, the excess packetsare discarded.

The queue regulator has to consider a number of factors when decidingwhether a packet of size P tokens can be forwarded into the network. Ifthe token bucket is full, the packet is forwarded into the network and Ptoken are removed from the bucket. If the token bucket is empty thepacket waits at the head of the queue until P tokens are generated andplaced in the bucket. When the bucket eventually contains P tokens, thatmany tokens are removed from the bucket and the packet is sent into thenetwork. Finally assume that the token bucket is only partially full andcontains T tokens. If P is less than or equal to T, P tokens are removedfrom the bucket and the packet is forwarded into the network. If P isgreater than T, the packet must wait for the remaining P minus T tokensbefore it can be sent into the network. When the bucket contains therequired P tokens, that many tokens are removed from the bucket and thepacket is forwarded into the network.

Token bucket rate-limiting tools can used to regulate the long-termaverage transmission rate while still permitting bursts of apredetermined size. The rate of the token generator defines thelong-term average traffic rate, the depth of the token bucket definesthe maximum burst size that the shaper allows, and the length of thepacket queue bounds the amount of delay that a packet can incur at thistraffic shaper.

Traffic Policing

Traffic-policing tools allow subscriber traffic flows to be examined andpackets that exceed SLAs to be either discarded or marked. Thetraffic-policing function uses a token bucket algorithm, but the packetqueue is replaced with a packet discard or a packet marking function. Ifthe policing function determines that a particular packet is in-profile,the packet is admitted to the network. If the policing functiondetermines that the packet is out-of-profile, the packet is eitherdropped immediately (hard policing) or admitted to the network butmarked as out-of-profile (soft policing).

Marking out-of-profile traffic (FIG. 31) allows in-profile andout-of-profile packets to be handled differently at hops downstream fromthe router performing the traffic-policing function. For example, atraffic-policing access router can be configured to mark a packet bychanging its drop precedence so that core routers give the packet anelevated discard precedence during periods of congestion while theycontinue to deliver in-profile traffic.

When using a traffic policer to mark packets, the policer should beconfigured to maintain packet ordering within a flow. Marking a packetby increasing its drop precedence simply raises the probability that acore router will drop the packet during periods of network congestion.Packet ordering is preserved because all packets in the flow areassigned to the same queue at each hop across the networks.

Marking a packet by assigning it to a different transmission queue basedon the in-profile or out-of-profile decision of a traffic policer shouldbe avoided. Separating a single traffic flow across multiple queues canresult in packet reordering that can negatively impact the flow'sthroughput performance. The probability that packet reordering willoccur is increased when the divided flow traverses multiple hops (withdifferent queues at each hop) across the core of the network.

Rate limiting using traffic-shaping tools has demonstrated its value byprecisely controlling the rate and volume of traffic as it enters thenetwork, but these tools do not support dynamic feedback mechanisms thatallow them to adapt to, and take advantage of, temporarily availablecapacity in the network core. Rate limiting using traffic-policing toolsprovides a more flexible approach than traffic shaping because it allowsresource protection schemes to adapt to changing network conditions.

While hard traffic policing (packet dropping) is similar to trafficshaping, soft traffic policing (packet marking) is a fundamentallydifferent approach. The ability to defer the decision to drop a packetto systems residing in the interior of the network allows traffic to bediscarded only when congestion actually exists in the core of thenetwork. During periods of abundant bandwidth, soft traffic policingallows service providers to deliver higher levels of service to all itssubscribers, while still providing the ability to protect sharedresources and meet subscriber SLAs during periods of scarce bandwidth.

Hard and Soft Traffic-Policing Applications

Hard and soft traffic-policing tools can be deployed at the edges of thenetwork to deliver tiered subscriber services and protect sharedresources. FIG. 32 illustrates how these tools can be used to limit theflow of subscriber traffic into the core of your network and manageoutbound bandwidth to subscribers or other service providers.

Because of the fundamental nature of Internet applications, asymmetricaltraffic flows are often required to be handled at the edges of the CPEnetwork. For example, content consumers accessing the Web typicallyrequires 10 times as much pull (download) bandwidth as push (upload)bandwidth. On the other hand, content providers need more push bandwidththan pull bandwidth. Traffic policing provides the tools to preciselycontrol bandwidth and deliver tiered SLAs based on specific subscriberor application requirements.

Traffic policing can be used to provision a subrate access circuit overa T1 (or E1) interface. For example, traffic policers can be configuredto support asymmetric traffic loads by offering 256 Kbps in thesubscriber push direction and 512 Kbps in the subscriber pull direction.Traffic policing can be used on a logical interface to apply a singlerate limit to all packets in the traffic stream. For example, trafficpolicing can be used to create a tiered VPN service based on the amountof bandwidth delivered to the subscriber's logical interface. Trafficpolicing can be used to rate-limit flows based on pet-class SLAs withina logical interface. For example, on a subscriber DS3 (45-Mbps) accesscircuit, 2 Mbps of traffic marked with a gold DiffServ Code Point valuecan be allowed, 5 Mbps of traffic marked with a silver DiffServ CodePoint value, and an unlimited amount of traffic marked with a bronzeDiffServ Code Point value. Traffic policing can be used to rate-limitflows based on a Layer 4 profile within a logical interface. Forexample, an unlimited amount of subscriber HTTP traffic can be allowed,but NNTP traffic to some bandwidth threshold can be rate limited.Traffic policing can be used to rate limit ICMP traffic to protectagainst denial-of-service (DoS) attacks. For example, a sniurf attackcan result in a large number of ICMP Echo Reply packets beingtransmitted to the target of the attack, This can result in severenetwork congestion or outages. Traffic policing can be used incombination with other packet-filtering tools to protect networkresources from these types of DoS attacks. Packet filtering (withoutrate limiting) can be used within a logical interface to queue trafficbased on a Layer 4 profile. For example, SMTP traffic can be assigned toa low-priority queue and assign VoIP traffic to a high-priority queue.

When evaluating the appropriateness of deploying traffic-policing toolsin different regions of a service provider network, a number of factors,for example, the granularity of the implementation's packetclassification tools, may be considered. If traffic-policing mechanismsare fully integrated with fine-grained packet-filtering tools, preciselyclassifying and regulating traffic flows based on a detailed examinationof packet headers should be considered. Another example is theperformance impact of enabling traffic-policing features. If thetraffic-policing function executes in hardware, a predictable impact onrouter forwarding performance can be expected. This allows trafficpolicing to be deployed in the network without the performancedegradation usually associated with conventional software-basedsolutions.

Asic-Based Packet Processor: Hard and Soft Traffic Policing OperationalModel

The Layer 3 hard and soft traffic-policing tools supported by theASIC-based packet processor perform their tasks using three distinctprocessing components (FIG. 33): Packet classifier, Flow monitor, andForwarding action.

The packet classifier examines the packets in a stream of packets anddivides them into policing equivalence classes (PECs). A PEC is a set ofpackets that is treated the same by the packet classifier. For example,consider a packet classifier that classifies based on a packet's IPdestination address and DiffServ byte. For simplicity, assume that thepacket classification table contains only the three rules shown in FIG.34.

In this example, consider a pair of packets, Packet A and Packet B.Assume that Packet A has an IP destination address 192.100.5.13 and aDiffserv byte value of 101110. Assume that Packet B has an IPdestination address 192.100.34.102 and a Diffserv byte value of 101110.Packet A and Packet B are in the same PEC (PEC 2) because they bothmatch the classification table according to rule 2. Consider a thirdpacket, Packet C, that has an IP destination address, 208.167.169.55,and a DiffServ byte value of 010100. Packet A and Packet C are not inthe same PEC, because Packet A matches rule 2 and Packet C matchesrule 1. Rule 0 is the default rule which matches each packet that doesnot match any other rule in the packet classification table.

The flow monitor maintains state concerning the limits on the load thata given PEC can place on the network. The flow monitor tracks twocritical parameters for each PEC, the average bandwidth and the maximumburst size. The values configured for each of these limits can bedifferent for each PEC. The flow monitor for each PEC performsaccounting operations on each packet to determine whether a packet isin-profile or out-of-profile.

The forwarding action for each packet assigned to a PEC depends onwhether the packet is in-profile or out-of-profile. If the packet isin-profile, it is simply accepted and forwarded without being affectedby the rate limiter or traffic policer. If the packet is out-of-profile,it can be either discarded or marked. A packet is marked by setting thePLP bit in the packet notification record to cause it to be processeddifferently by the packet processing engines downstream from the trafficpolicing function. The term downstream can refer to either a process inthe local router or a subsequent router in the packet's path across thenetwork.

ASIC-Based Packet Processor: Traffic-Policing Configuration EXAMPLES

Each traffic stream has up to four output transmission queues, numbered0 through 3, with each queue receiving a configured percentage of thestream's available queue buffer size (FIG. 35). A router determines theoutput queue for each packet by reading the value of the two queuepriority bits (00, 01, 10, 11) carried in the packet notificationrecord.

While the WRR algorithm services the output queues, a RED algorithmconstantly monitors each queue for congestion. If a packet is out ofprofile, the RED algorithm can use a more aggressive drop probabilitythan if the packet is in-profile. A router determines whether eachpacket is in-profile (PLP=0) or out-of-profile (PLP=1) by reading thepacket loss priority (PLP) bit carried in the packet notificationrecord.

Finally, a router executing the policing function can be configured towrite the PLP bit to cause the local router to select a specific REDdrop profile. Additionally, the router executing the policing functioncan be configured to rewrite the RED drop priority bit in the IPprecedence bits before transmitting the packet. By rewriting the IFPprecedence bits before transmitting a packet, the policing router caninstruct downstream routers to use a specific RED drop profile when itis queues marked packets.

Specifying the Traffic Policer

Traffic policing allows the amount of traffic that passes into or out ofa router interface to be limited. Two types of limits to packets in aPEC can be applied: bandwidth limit, which defines the average number ofbits permitted per second, and burst size limit, which defines the totalnumber of bytes per second over and above the bandwidth limit allowed ina burst.

The general format for specifying a traffic policer is given by thefollowing syntax:

policer sample-policer { if-exceeding bandwidth-limit value;burst-size-limit value; } then { /* Execute if and only if (iff) trafficis out of profile */ } }

The unit for the bandwidth-limit is expressed in bits per second (bps),while the unit for the burst-size-limit is expressed in bytes. Thesuffixes k, m, and g can be used to denote multipliers of one thousand(kilo), one million (mega), and one billion (giga), respectively, Forexample, suppose it is desirable to specify a traffic policer with abandwidth limit of 20 megabits per second (Mbps) and a burst size limitof 125 kilobytes (KB). This can be achieved with the followingconfiguration:

policer sample-policer { if-exceeding C bandwidth-limit 20m;burst-size-limit 125k; } then { /* Execute iff traffic is out of profile*/ } }

The possible actions for a traffic policer when the packet isout-of-profile are to either discard the packet or mark the packet. Whenmarking a packet, the PLP bit can be written to select the RED dropprofile for a packet. The PLP bit can have a value of 0 or 1. Forexample, to discard an out-of-profile packet, the followingconfiguration can be used:

policer sample-policer { if-exceeding { bandwidth-limit 20m;burst-size-limit 125k; } then { discard; } }

To accept an out-of-profile packet and set the PLP bit to 1, thefollowing configuration can be used:

policer sample-policer { if-exceeding { bandwidth-limit 20m;burst-size-limit 125k; } then { plp 1; accept; } }

It is not necessary to explicitly configure a counter for out-of-profilepackets in the then clause of a traffic policer specification. Thesoftware automatically maintains out-of-profile statistics for eachpolicer specification.

Example: Rate Limiting Using a Coarse-Grained Hard Traffic Policer

This example illustrates how you can use a coarse-grained, hardtraffic-policing filter to deliver rate-limited services at an interfacegranularity. For example, hard traffic policing can used to provision a256-kbps subscriber access service on a TI (or E1) circuit. When thesubscriber desires to increase its bandwidth to 512 Kbps, the hardtraffic policing configuration parameters can simply be changed to grantadditional bandwidth to the subscriber.

Performing rate limiting using Layer 3 hard traffic policing rather thanconventional Layer 1 DSU configuration simplifies service creation,operation, and support. Coordinating the reconfiguration of layer 1 DSUsat both ends of the access circuit is difficult and time consuming. Inmany cases, doing this requires extremely close coordination between theservice provider that delivers the IP service and the carrier thatprovides the access circuit. Also, until the DSUs at both ends of theaccess circuit are reconfigured, the circuit cannot become operational.Increasing subscriber bandwidth requires only the reconfiguration of thetraffic policer on the provider access router, rather than thecomplexity and expense of reconfiguring multiple devices. Unlike thetraditional DSU configuration model, rate limiting using Layer 3 trafficpolicing does not have to be symmetrical. A traffic policer with aspecific rate limit for traffic flowing from the subscriber into thenetwork can be configured. A traffic policer with a differentiate limitfor traffic flowing from the network to the subscriber can also beconfigured.

The filter in this example delivers a substrate service by implementinga hard traffic policer for all packets arriving at interface t1-1/0/0 ofthe provider access router (see FIG. 36). If the policer determines thata subscriber packet is out-of-profile, the packet is discarded. If thepolicer determines that a packet is in-profile, the packet is countedand accepted.

This example illustrates a number of details about the configuration andoperation of packet filters that invoke the traffic-policing function.Once a policer is specified, it can be referenced in the then clause ofa filter term. The standard actions permitted in the then clause of afilter term (such as accept, discard, count, and log) are also permittedin the then clause of a policer specification. The traffic policer isapplied to each packet that matches a filter term before the otheractions in the then clause of the filter term are executed. If thepolicer determines that a packet is out-of-profile, the actions in thethen clause of the policer specification term are executed and those inthe then clause of the filter term that invoked the policing functionare not executed. If the policer determines that a packet is in-profile,the actions in the then clause of the of the filter term that invokedthe policing function are executed, and those in the then clause of thepolicer specification term are not executed.

The following configuration specifies the coarse-grained, hardtraffic-policing filter:

filter coarse-hard-policer { policer monitor-subscriber { /* Specifypolicer */ if-exceeding {, bandwidth-limit 256k; /* Bandwidthlimitations */ burst-size-limit 2k; } then { /* Execute iff out ofprofile */ discard; /* Discard out of profile */ } } termrate-limit-subscriber { /* Match all traffic */ then { policermonitor-subscriber; /* Invoke policer */ count in-profile; /* Count inprofile */ accept; /* Accept in profile */ } } }

The following configuration assigns this filter to input interfacet1-1/0/0 on the provider access router:

interfaces t1-1/0/0 { unit 0 { family inet { filter { inputcoarse-hard-policer; } } } }

The policer specification monitor-subscriber defines the parameters forthe traffic policer, specifying a bandwidth limit of 256 kbps and aburst size of 2 KB. If a filter term invokes monitor-subscriber and thepolicer determines that the packet is out-of-profile, the packet issilently discarded.

The filter term rate-limit-subscriber matches all packets in the trafficstream because the filter term does not contain a from clause. If thepolicer monitor-subscriber determines that a matching packet isout-of-profile, the packet is silently discarded. If the policerdetermines that a matching packet is in-profile, the counter in-profileis incremented and the packet is accepted.

Example: Rate Limiting Using a Fine-Grained Hard Traffic Policer

This example illustrates how a fine-grained, hard traffic-policingfilter can be used to rate limit a specific subscriber application. Thefilter in this example polices all HTTP traffic that arrives at provideraccess router interface t1-1/0/0 and is addressed to server 221.5.6.7(see FIG. 37). If the policer determines that the packet isout-of-profile, the packet is discarded. If the policer determines thatthe packet is in-profile, the packet is counted and accepted.

The following configuration specifies the fine-grained, hardtraffic-policing filter:

filter fine-hard-policer {  policer monitor-http { /* Specify policer */  if-exceeding {,    bandwidth-limit 155k; /* Bandwidth limitations */   burst-size-limit 1k;   }   then { /* Execute iff out of profile */   discard; /* Discard out of profile */   }  }  term rate-limit-http {/* Police all HTTP to server */   then {    policer monitor-subscriber;/* Invoke policer */    count in-profile; /* Count in profile */   accept; /* Accept in profile */   }  }  term default-action { /*Accept all traffic not */   then accept; /* matching a filter term */  }}

The policer specification monitor-http defines the parameters for thetraffic policer, specifying a bandwidth limit of 155 kbps and a burstsize of 1 KB. If a filter invokes monitor-http and the policerdetermines that the packet is out-of-profile, the packet is silentlydiscarded. The filter term rate-limit-http-server examines all packetsin the stream and matches those addressed to the HTTP server. If thepolicer monitor-to-http determines that a matching packet isout-of-profile, the packet is silently discarded. If the policerdetermines that a matching packet is in-profile, the counterhttp-in-profile is incremented and the packet is accepted. The filterterm default-action accepts all packets that do not match the packetclassification conditions of the rate-limit-http-server filter term.

Example: Rate Limiting Using a Fine-Grained Soft Traffic Policer

Rate-limiting subscriber traffic can be achieved by deploying hardtraffic-policing filters that simply drops out-of-profile packets.Rate-limiting subscriber traffic may be achieved by deployingfine-grained, soft traffic-policing filters that defer the drop decisionto downstream routers by marking the RED drop profile for specificpackets. To control the behavior of downstream routers in the core ofthe network, any changes made to the packet notification record by arate-limiting, traffic-policing filter should be rewritten to the IPprecedence bits before the packet is transmitted into the core of thenetwork.

This example describes how to rate-limit subscriber traffic using a softtraffic-policing filter within the context of a single service queue andmarking packets with either a low or a high priority RED drop profile.The primary benefit of this approach is that it does not require themanagement complexity of enabling DiffServ on all systems across thenetwork. Although DiffServ will play an increasingly important role asthe Internet continues to mature, service providers are still trying tounderstand how to best market, deploy, bill, operate, inter-network, andperform capacity planning for Diffserv networks.

IP Precedence Rewrite Configuration

Before examining the soft traffic-policing filter, consider how thesoftware may allow an access router to be configured to rewrite the IPprecedence bits before a packet is transmitted to the next-hop router.This is accomplished by configuring CoS to specify exactly how the bitsare rewritten when the packet notification record is removed from theoutput queue and the referenced packet is reassembled from the sharedmemory architecture. FIG. 38 illustrates the following CoS configurationthat rewrites the IP precedence bits and assigns the new configurationto output interface so-6/0/0:

class-of-service { output { interfaces { so-6/0/0 { unit 0 {precedence-rewrite { output-queue 0 { plp-clear rewrite-bits 000;plp-set rewrite-bits 001; { } } } } } }

The default configuration for a router is to support a single queue,queue 0, on each output interface. If the PLP bit in the packetnotification record removed from queue 0 is cleared to 0, the IPprecedence bits in the referenced packet are rewritten to 000. Thesebits inform downstream routers that the packet should be assigned toqueue 0 with the PLP bit cleared to 0. If the PLP bit in the packetnotification record removed from queue 0 is set to 1, the IP precedencebits in the referenced packet are rewritten to 001. These bits informdownstream routers that the packet should be assigned to queue 0 withthe PLP bit set to 1.

The filter in this example polices all HTTP traffic that arrives atprovider access router interface t1-1/0/0 and is addressed to server221.5.6.7 (see FIG. 39). If the policer determines that the packet isout-of-profile, the IP precedence bits are rewritten so downstreamrouters give it a higher priority RED drop profile. If the policerdetermines that the packet is in-profile, the IP precedence bits arerewritten so downstream routers give the packet a lower priority REDdrop profile.

Assume that a provider access router has the following configuration:

interfaces t1-1/0/0 { /* Assign filter to input interface */   unit 0 {    family inet {       filter {         input sample-rewrite;       }    }   } } class-of-service { /* Define IP precedence rewrite */  output {     interfaces {       so-6/0/0 {         unit 0 {          precedence-rewrite {             output-queue 0 {              plp-clear rewrite-bits 000; /* If PLP=0 */              plp-set rewrite-bits 001; /* If PLP=l */             {          }         }       }     }   } } filter sample-rewrite {  policer monitor-http { /* Specify policer */     if-exceeding {      bandwidth-limit 155k; /* Bandwidth limitations */      burst-size-limit 1k;     }     then { /* Execute iff out ofprofile */       plp 1; /* If out of profile, PLP =1 */       accept; /*Accept out of profile */     }   }   term police-to-http-server { /*Police HTTP traffic */     from {       destination-address 221.5.6.7;      protocol tcp;       destination-port http;     }     then {      policer monitor-http; /* Invoke policer */       p1p 0; /* If inprofile, PLP = 0 */       accept; /* Accept in profile */     }   }  term default-action { /* Accept all traffic not */     then accept; /*matching a filter term */   } }

The interfaces t1-1/0/0 statement assigns the sample-rewrite packetfilter to input interface t1-1/0/0. The class-of-serviceprecedence-rewrite statements specify how the IP precedence hits are tobe rewritten before a packet is transmitted on interface so-6/0/0 to thenext-hop router. For a packet notification record emerging from the headof queue 0 with a PLP=0, the configuration causes the IP precedence bitsin the referenced packet to be rewritten to 000 (queue=0, PLP=0). For apacket notification record emerging from the head of queue 0 with aPLP=1, the configuration causes the IP precedence bits in the referencedpacket to be rewritten to 001 (queue=0, PLP=1).

The sample-rewrite filter classifies and appropriately marks packetsbased on packet header fields. The policer specification monitor-httpdefines the parameters for the traffic policer, specifying a bandwidthlimit of 155 kbps and a burst size of 1 KB. If a filter term invokesmonitor-http and the policer determines that the packet isout-of-profile, the PLP bit is set to 1 and the packet is accepted.

The filter term, police-to-http-server, examines all packets in thestream and identifies the packets addressed to the HTTP server. If thepolicer monitor-http determines that the matching packet isout-of-profile, the PLP bit is set to 1 and the packet is accepted. Whenthe packet notification record for the out-of-profile packet is removedfrom queue 0, the referenced packet has its IP precedence bits rewrittento 001 before it is transmitted to the next-hop router. If the policerdetermines that the matching packet is in-profile, the PLP bit iscleared to 0 and the packet is accepted. When the packet notificationrecord for the in-profile packet is removed from queue 0, the referencedpacket has its IP precedence bits rewritten to 000 before it istransmitted to the next hop-router.

The filter term, default-action, accepts all packets that do not matchthe packet classification conditions of the term, police-to-http-server,and allows them to be accepted by the packet filter.

This example includes steps to configure a soft traffic policer on arouter deployed at the edge of the network. The traffic policer rewritesthe drop priority bit in then precedence bits to communicate thein-profile or out-of-profile decision made at the access router todownstream routers in the core of the network.

Example: Queue Selection Using a Fine-Grained Packet Filter

Packets may be assigned to an output transmission queue based on anexamination of packet header fields. The filter in this exampleidentifies all HTTP packets that arrive on provider access routerinterface t1-1/0/0 and are addressed to server 221.5.6.7. If a packetmatches the filter, the packet is assigned to output queue 1 (FIG. 40).

The following configuration specifies the fine-grained, packetclassification filter:

filter sample-filter {  term queue-http { /* Queue HTTP traffic */  from {    destination-address 221.5.6.7;    protocol tcp;   destination-port http;   }   then {    output-queue 1; /* Assign allHTTP to Queue 1 */    accept; /* Accept HTTP packets */   }  }  termdefault-action { /* Accept all traffic not */   then accept; /* matchinga filter term */  } }

The filter term, queue-http examines all packets in the traffic stream.If the packet is HTTP and addressed to server 221.5.6.7, the queuepriority bits in the packet notification records are set to 1, causingthe packet to be assigned to queue 1 on the output interface. The filterterm, default-action, accepts all packets that do not match the filterterm queue-http and allows them to be accepted by the filter.

This filter does not cause packet reordering because all HTTP packetsmatching the filter conditions are assigned to queue 1 in the localrouter. However, the configuration in this example is incomplete becausethe configuration does not rewrite the queue priority bits in the IPprecedence bits to communicate the queue selection to downstream routersin the core of the network. To complete this configuration, one needs toconfigure multiple queues on the access router, configure CoS to rewritethe IP precedence bits on the output interface of the access router, andconfigure core routers to read the IP precedence bits and assign packetsto the proper output transmission queue.

Other embodiments are within the scope of the following claims.

1. A method performed by one or more devices, the method comprising:filtering, by the one or more devices, received packets based on afilter rule associated with the received packets; marking, for sampling,by the one or more devices, a first one of the received packets bysetting a bit in a packet notification when the first received packetmatches the filter rule; and sampling, by the one or more devices, themarked packet based on a randomly generated number associated with themarked packet.
 2. The method of claim 1, where the sampling is performedbased on a comparison of the randomly generated number to apredetermined threshold.
 3. The method of claim 1, where the receivedpackets include header information comprising at least one of: a sourceInternet protocol (IP) address, a destination IP address, an IP type, asource port, a destination port, a differentiated service (DiffServ)byte, an IP fragmentation offset field, an IP fragmentation controlfield, or a transmission control protocol (TCP) control bit, and wherethe filtering is based on the header information.
 4. The method of claim1, further comprising at least one of: accepting the marked packet;discarding the marked packet; or rejecting the marked packet andtransmitting an Internet control message protocol (ICMP) message.
 5. Themethod of claim 1, where the filtering comprises accepting an unmarkedpacket when the unmarked packet is not rejected based on the filterrule.
 6. The method of claim 1, further comprising: writing headerinformation associated with the sampled packet to a routing engine ofthe one or more devices.
 7. The method of claim 1, further comprising atleast one of: monitoring all logical interfaces associated with the oneor more devices; monitoring designated logical interfaces associatedwith the one or more devices; monitoring designated protocols associatedwith the received packets; monitoring a range of addresses associatedwith the received packets; or monitoring individual addresses associatedwith the received packets.
 8. The method of claim 1, where the filteringidentifies at least one of respective destinations of the receivedpackets, a volume of the received packets, or respective contents of thereceived packets.
 9. The method of claim 1, further comprising: loggingthe marked packet, a log entry associated with the logged packet beingaccessible for display via an interface associated with the one or moredevices, the log entry including at least one of a log time, an inputcircuit, a protocol type, a source address, or a destination address.10. The method of claim 1, where the filtering comprises performingsource address verification to prevent source address spoofing of anetwork operation center system.
 11. The method of claim 1, furthercomprising: load balancing the received packets.
 12. The method of claim11, where the load balancing comprises forwarding received packetsassociated with a designated source port or a designated source addressto a designated destination port or a designated destination address.13. The method of claim 12, where the forwarding the received packetscomprises maintaining an order and a travel path for a TCP sessionassociated with the forwarded packets.
 14. A method performed by one ormore devices, comprising: randomly sampling, by the one or more devices,received packets based on a predetermined sampling rate of 1/n, where nis a specified integer; comparing, by the one or more devices, thepredetermined sampling rate to a randomly generated number to form aresulting value; and performing, by the one or more devices, the randomsampling based on the resulting value.
 15. A system comprising one ormore devices to: accept a packet, when the packet is not rejected by afilter, when a filtering criterion is satisfied, by setting a bit in apacket notification; and mark the accepted packet for sampling when arandomly generated number is less than a predetermined value.
 16. Thesystem of claim 15, where the one or more devices are further to:discard the packet when the packet contains a spoofed network operationcenter source address.